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Abstract 


The  recently  developed  NATO  NEC  Command  and  Control  Maturity  Model  (N2C2M2)  defined  new  C2  approaches, 
established  their  relation  with  one  or  more  maturity  levels  and  studied  the  implications  throughout  the  NCW  value 
chain,  the  C2  domains  (information,  cognitive  and  social)  and  the  organization’s  effectiveness. 

This  paper  focus  on  a  set  of  experiments  especially  developed  to  validate  N2C2M2’s  hypothesis  and  implications  by 
instantiating  each  of  the  models’  C2  approaches,  applying  ELICIT,  a  network-enabled  collaborative  environment 
involving  17  human  subjects,  and  conduct  a  quantitative  analysis  on  several  C2  domains  and  variables. 

The  experiments  confirmed  significant  differences  among  the  several  C2  approaches,  demonstrating  that  increasing 
the  C2  approach  maturity  of  an  organization,  increases  the  extent  of  shared  information  and  critical  information 
available,  result  in  better  interactions,  broader  extent  of  correct  understanding,  self-synchronization  and  increased 
organizational  effectiveness  and  efficiency.  More  specifically,  COLLABORATIVE  approach  was  the  most  effective 
and  efficient,  while  EDGE  approach  performed  best  in  making  critical  information  available  to  the  right  positions, 
reached  highest  extent  of  correct  understanding  and  most  ordered  state  of  cognitive  self-synchronization.  As  expected, 
CONFLICTED  approach  achieved  worst  scores  in  all  domains. 
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INTRODUCTION 

Information  Age  brought  about  new  and  powerful  tools.  In  particular,  the  capability  to  exchange  and  process  huge 
amounts  of  information  in  real-time  between  remote  and  distant  parties  has  led  to  the  development  of  new  Command 
and  Control  (C2)  approaches,  such  as  Network  Centric  Operations  (NCO)  and  Power-to-the-Edge  principles  (Alberts 
and  Hayes  2003),  being  the  latter  characterized  by  broad  distribution  of  information,  unconstrained  patterns  of 
interaction  and  dynamic  allocation  of  decision  rights. 

In  fact,  C2  capabilities  have  more  to  do  with  the  largely  unconsolidated  theoretical  framework  that  must  lie  beneath 
the  way  information  domain  is  employed  and  used  by  individuals  in  order  to  generate  the  desired  emergent  behaviors 
in  the  cognitive  and  social  domains,  supporting  the  development  of  shared  intent,  shared  awareness  and 
understanding,  which  in  turn  leads  to  synchronized  actions  to  achieve  the  desired  effects.  The  definition  of  that 
theoretical  framework  represents  a  long  way  ahead  and  it  is  expected  that  experimentation  will  play  a  key  role  for  its 
development  and  validation. 

Notable  recent  work  has  been  conducted  in  this  field.  C2  Conceptual  Reference  Models  (C2  CRM)  were  developed  by 
the  CCRP  ASD-NII/OFT  (Alberts  and  Hayes  2006)  and  NATO  SAS-050  (SAS-050  2006),  providing  useful  and 
innovative  insights  towards  the  most  important  variables  that  characterize  C2  processes  and  their  interrelations. 
Another  effort  was  taken  by  NATO  SAS-065  to  develop  the  NATO  NEC  C2  Maturity  Model  (N2C2M2)  (SAS-065 
2010),  associating  the  ability  to  adopt  one  or  more  approaches  to  C2  with  different  levels  of  C2  maturity  and 
describing  its  impact  on  numerous  variables  present  in  the  C2  CRM. 

This  paper  presents  main  findings  of  a  recent  effort  undertaken  to  validate,  through  experimentation,  the  recently 
developed  N2C2M2,  while  at  the  same  time,  understanding  the  implications  of  its  C2  approaches,  framed  in  terms  of 
the  NCW  (Alberts  et.  al.  1999)  (Alberts  and  Hayes  2006)  and  NATO  (SAS-050  2006)  (SAS-065  2010)  conceptual 
frameworks. 

This  paper  is  organized  as  follows:  it  starts  with  a  description  of  relevant  Background  in  terms  of  theory  (including 
fundamentals  of  NCW  and  the  N2C2M2)  and  research  (mainly,  the  ELICIT  platform),  then,  in  “Experiment 
Hypothesis  and  Early  Expectations”,  it  enumerates  the  hypothesis  to  test,  the  experimentation  model  (with  explicit 
presentation  of  independent  and  dependent  variables  and  their  interrelations)  and  the  experimental  design,  in 
“Experiments  Analysis”  it  presents  the  analysis  of  the  experiments  and  it  finishes  with  the  conclusions. 

Given  the  large  extent  of  this  work,  an  Annex  is  included  with  this  paper  to  present  detailed  information  about  the 
N2C2M2  experimentation,  ELICIT  platform  and  detailed  data. 


BACKGROUND 

Most  of  this  work’s  foundations  are  the  fundamental  aspects  of  Network-Centric  Warfare  (NCW)  including  the  NCW 
tenets1 ,  NCW  Value  Chain  (SAS-065  2010,  27),  C2  Domains  (Alberts  and  Hayes  2006),  C2  CRM  (SAS-050  2006) 
(Alberts  and  Hayes  2006)  and  C2  Approach  Space  (SAS-050  2006)  (Alberts  and  Hayes  2006). 

These  foundations  were  used  to  define  the  experimentation  model,  key  variables,  their  interrelations  and  design.  A 
core  aspect  encompassing  this  work  consisted  in  enabling  the  observation  of  several  C2  CRM  variables  and  their 
consequences  as  expressed  by  the  NCW  Value  Chain ,  namely,  observing  how  changing  the  way  a  force  is  networked 
(i.e.,  its  C2  approach)  affects  (the  quality  of)  shared  information  and  collaboration,  and  in  turn  impacts  shared 
situational  awareness,  shared  understanding  and  mission  effectiveness. 

Recent  experimentation  efforts  within  the  context  of  NCW  and  C2  approaches  have  been  conducted  through  ELICIT, 
a  research  and  experimentation  programme  developed  for  the  CCRP,  to  better  understand  the  issues  of  collaboration, 
information  sharing  and  trust  (see  “Overview  of  the  ELICIT  Platform”  in  Annex  and,  for  detailed  information,  (Ruddy 
2007)).  ELICIT  includes  a  game -based  software  simulation  platform  that  provides  a  network-centric  collaborative 
environment  for  a  small  group  or  organization  (constituted  by  humans  and/or  software  agents),  allows  instantiation  of 
different  C2  approaches  and  enables  observation  of  behaviors  and  dynamics  in  the  information,  cognitive  and  social 
domains  due  to  its  extensive  and  unobtrusive  logging  capabilities. 


1  Network  Centric  Warfare  Department  of  Defense  Report  to  Congress.  July  2001. 
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ELICIT  was  originally  developed  to  test  hypothesis  related  with  edge  and  hierarchical  (traditional)  command  and 
control  practices  (Ruddy  2007)  and  its  increased  configuration  capabilities  and  flexibility  has  enlarged  its  fields  of 
application  for  research. 

More  specifically,  the  ELICIT  web-version  (webELICIT)  (Ruddy  2008)  will  be  used  as  a  network-centric 
experimentation  platform  for  this  work  since  it  provides  easy  manipulation  and  setup  of  organization  models,  control 
of  communications,  and,  more  importantly,  a  clear  mapping  with  the  theory  of  NCW,  including  a  subset  of  the  C2 
CRM  where  several  variables  of  interest  are  observable,  including:  Quality  of  Individual  and  Shared  Information 
Position,  Information  Distribution,  Patterns  of  Interaction,  Quality  of  Individual  and  Shared  Understanding,  Quality 
of  Interactions,  Self-Synchronization,  Mission  Effectiveness  and  Mission  Efficiency  (given  Effectiveness)  (Manso  and 
Nunes  2007)  (McEver,  Hayes  and  Martin  2007)  (Martin  and  McEver  2008). 

A  preliminary  experimentation  effort,  conducted  within  the  aegis  of  NATO  SAS-065,  mapped  ELICIT-Hierarchy 
model  as  an  approximation  of  De-Conflicted  C2  while  ELICIT-Edge  model  to  a  region  of  the  C2  approach  space 
further  along  the  central  diagonal  vector  (toward  Edge  C2)  allowing  testing  the  hypothesis  that  More  Mature  Levels  of 
C2  would  Perform  More  Efficiently  and  More  Effectively.  Results  were  clear  and  unambiguous  (SAS-065  2010,  132): 
Edge  organizations  were  more  effective,  faster,  shared  more  information  and  were  more  efficient  than  Hierarchies. 
Also  based  on  empirical  evidence,  previous  research  work  was  also  consistent  with  the  hypothesis  that  Edge 
outperformed  Hierarchies  (Ruddy,  2007)  (McEver,  Hayes  and  Martin  2007)  (Martin  and  McEver  2008).  This  work 
attempts  to  go  further  by  recreating  the  N2C2M2  five  C2  approaches  in  the  ELICIT  platform  and  observe  its  outcomes 
and  main  differentiating  aspects. 

Finally,  we  are  fully  aware  of  the  ELICIT  oversimplification  towards  application  of  C2  processes,  nature  of  the 
problem  (tame  and  well  defined)  and  limitation  on  the  collaboration  level  between  subjects  (e.g.,  fixed  text  share  and 
post  actions).  These  conditions  only  limit  the  extent  to  which  ELICIT  may  validate  the  model. 

The  N2C2M2  is  briefly  introduced  next. 

The  NATO  NEC  C2  Maturity  Model  (SAS-065  2010) 

The  N2C2M2  defines  several  degrees  of  operational  coherence  (the  ability  to  generate  synergies  across  a  set  of 
participants)  that  can  be  achieved.  These  are  framed  into  the  five  levels  of  NATO  NEC  operational  capability  (levels  1 
to  5).  Associated  with  each  level  is  the  ability  of  the  collective  to  adopt  one  or  more  approaches  to  C2.  Moreover, 
associated  with  increased  maturity  is  the  ability  to  adopt  a  wider  range  of  approaches  to  C2  that,  in  turn,  cover  a  large 
portion  of  the  C2  Approach  Space. 

The  five  classes  of  Collective  C2  Approaches  defined,  representing  a  major  differentiating  aspect  in  each  maturity 
level,  are:  Conflicted  C2;  De-conflicted  C2;  Coordinated  C2;  Collaborative  C2;  and  Edge  C2. 

These  approaches  fit  into  specific  regions  of  the  Collective  C2  approach  space2,  as  depicted  in  Figure  1. 


2  Note  that  the  N2C2M2  deals  with  the  set  of  entities  engaged  in  a  complex  endeavour  (Alberts  and  Hayes  2009,  4).  Hence,  the  concept  of  C2  approach  is 
interpreted  in  the  perspective  of  a  ‘collective’  (i.e.,  Collective  C2).  This  implies  re-interpretation  of  the  dimensions  of  a  Collective  C2  approach  space  (SAS-065 
2009,  2)  as  allocation  of  decision  rights  to  the  collective  (ADR-C),  patterns  of  interaction  among  entities  (PI-C  ),  and  distribution  of  information  among  entities 
(DI-C). 
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& 


C2  Approaches 

- Edge  C2  □ 

--Collaborative  C2  □ 

—  Coordinated  C2  C 

-  De-Conflicted  C2  □ 
- Conflicted  C2  H 


Figure  1  -  Collective  C2  Approach  Space 


Higher  maturity  levels  include  the  ability  to  adopt  C2  approaches  located  at  the  ‘upper  right’  side  of  the  C2  approach 
space  (e.g.,  Collaborative  and  Edge).  For  convenience,  approaches  located  at  upper  right  comer  will  also  be  referred 
as  more  mature  than  those  located  at  lower-left  corner. 

The  qualitative  value  for  each  dimension,  per  C2  Approach,  is  presented  in  Table  1. 


ADR-C 

Pl-C 

Dl-C 

Not  Explicit  Self 
Allocated  (Emergent 
Tailored,  and  Dynamic) 

Unlimited  Sharing  as 
Required 

All  Available  and  Relevant 

Information  Accessible 

Collaborative  Process  and 

Shared  Plan 

Significant  Broad  Sharing 

Additional  Information 

Across  Collaborative 
Areas/Functions 

Coordination  Process  and 
Linked  Plans 

Limited  Focused  Sharing 

Additional  information 

about  coordinated 
Areas/Functions 

Establish  Constraints 

Very  Limited  Sharply 
Focused  Sharing 

Additional  information 

about  constraints  and 

seams 

None 

No  interactions  across 

entities 

Organic  information. 

Table  1  -  Qualitative  values  for  C2  Dimensions  per  Maturity  Level 


A  main  assumption  in  the  N2C2M2  is:  more  mature  Collective  C2  Approaches  (i.e.,  more  distribution  of  decision 
rights  across  the  collective,  less  constrained  patterns  of  interaction  and  broader  dissemination  of  information)  achieve 
higher  levels  of  shared  awareness  and  understanding  which,  in  turn,  result  in  increased  endeavor  effectiveness, 
efficiency  (given  effectiveness)  and  agility. 

This  is  summarized  in  Table  2  and  it  will  support  formulation  of  the  main  hypothesis  for  verification  addressed  in  this 
work  (see  next  section). 
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Edge  C2 


Collaborative  C2 


Coordinated  C2 


De-conflicted  C2 


Conflicted  C2 


Degree  of 
Shared 

Awareness 

Degree  of 
Shared 

Understanding 

Broad,  Deep, 
Tailored  and 
Dynamic 

Broad,  Deep, 
Tailored  and 
Dynamic 

Significant 

Significant 

Limited 

Limited 

Focused  on  the 

boundaries 

None 

None 

None 

Relative 

Effectiveness 

Efficiency,  Given 
Effectiveness 

Tailored  and 
dynamic  synergies 

Highly  efficient 

Substantial 
synergies  across 
collaborative 
areas/functions 

Substantial 

efficiencies  across 
collaborative 
areas/functions 

Limited  synergies 
due  to 

coordination 

Limited 

efficiencies  due  to 

coordination 

Avoids  costs  of 
negative  cross¬ 
impacts 

Sub-optimized 
use  of  resources 

Negative  cross¬ 
impacts 

Inefficiency 
wasted  resources 

Agility  of  the  Collective 
C2  Process 


Proactive  across  a  broad 
range  of  conditions 


Substantial,  timely  and 
continuous 


Limited  to  coordinated 
functions/actions;  Slow; 
Reactive 


Vulnerable  at  seams; 
Rigid  from  specialization 


Fragile  and  vulnerable 
at  the  seams 


Table  2  -  Results  of  C2  Approaches  over  MoM  and  MOFE 


EXPERIMENT  HYPOTHESIS  AND  EARLY  EXPECTATIONS 

The  hypotheses  for  validation  are: 

[1]  For  a  complex  endeavor3,  higher  collective  C2  maturity  approaches  are  more  effective. 

[2]  For  a  given  level  of  effectiveness,  higher  collective  C2  maturity  approaches  are  more  efficient. 

[3]  Higher  collective  C2  maturity  approaches  are  more  agile. 

Also  relevant  to  measure  and  analyze  in  the  same  context  are  intermediate  variables  associated  with  the  network 
centric  value  chain  for  they  may  provide  important  insights.  The  following  hypotheses  will  also  be  tested: 

Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of: 

[4]  Quality  of  Individual  and  Shared  Information; 

[5]  Quality  of  Individual  and  Shared  Awareness  and  Understanding; 

[6]  Self-Synchronization  (at  cognitive  level); 

Than:  lower  collective  C2  maturity  approaches. 

Finally,  it  is  expected  a  minimum  level  of  maturity,  that,  if  not  met,  an  organization  is  not  effective  in  the  ELICIT 
game,  i.e.,  requisite  maturity  (SAS-065  2010,  85)  for  the  ELICIT  game. 

[7]  Organizations  require  a  minimum  level  of  maturity  to  be  effective  in  ELICIT. 

[8]  Increasing  the  degree  of  difficulty  in  ELICIT  require  organizations  to  increase  their  level  of  maturity  to 

maintain  effectiveness  in  ELICIT. 


3  It  has  been  a  matter  of  some  debate  if  ELICIT  presents  a  complex  problem  and  if  it  achieves  complexity  at  a  systemic  level.  While  the  problem  enunciated  by 
ELICIT  may  be  classified  as  simple  or  tame  (i.e.,  is  well  defined  and  may  be  solved  by  elimination),  the  fact  is  that  produced  dynamics  to  reach  a  sufficient  state 
of  awareness  (e.g.,  effective  sharing  of  factoids  across  an  organization)  depend  heavily  on  subjects  behaviors  and  will  and  these  are  unpredictable  and  non- 
deterministic  (hence  complex).  The  fact  is  that,  against  expectations,  more  often  than  not  subjects  don’t  reach  sufficient  levels  of  distribution  of  information, 
awareness  and  understanding  and,  consequently,  effectiveness  in  ELICIT  is  usually  low  (see  SAS-065  2010). _ 
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Experimentation  Model 

The  developed  validation  model  for  these  experiments  is  depicted  in  Figure  2  and  is  an  adaptation  of  the  NCW  value 
chain  (Alberts  and  Hayes,  2003)  and  the  ELICIT  model  (Alberts  2009)  (SAS-065  2010,  219).  This  model  includes  the 
variables  of  interest  and  their  key  relationships. 


* 


Information  ^ 

Sharing  /  Collective  C2 

- - '*L  Approach 

Patterns  of  V',DR-C,  Pl-C,  Dl-C) 
Interaction 


Allocation  of 
Decision  Rights 


Network 

Info  Sharing  & 

Characteristics 
&  Performance 

W 

Collaboration 

Individual 

Quality  of 
Information 

Quality  of 
Awareness  & 
Understanding 

Quality  of 
Individual 
Decisions 

|  . . - . J 

r— 

Self- 

Synchronization 


Task  Effectiveness 
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Figure  2  -  N2C2M2  Model  for  Validation  in  ELICIT 


The  N2C2M2  key  independent  variable  is  Collective  C2  Approach ,  which  is  expressed  in  terms  of  its  three  C2  key- 
dimensions  variables:  ADR-C,  PI-C  and  DI-C  (SAS-065  2010).  The  experimentation  design  will  seek  to  influence 
and  control  these  three  variables,  considering  their  highly  dynamic  nature,  interrelations  and  dependency  on  humans’ 
free  will  to  interact  and  collaborate  (note  that  ELICIT  provides  an  interoperable  infrastructure).  Regarding  this  matter, 
we  consider  that,  whenever  humans  are  involved,  control  is  not  an  accurate  term  and  influence  should  be  used  instead. 
Hence,  experiments  are  designed  to  provide  (i)  initial  conditions  to  operate  at  a  specific  C2  Approach  (e.g.,  network 
connectivity,  network  performance)  and  (ii)  incentives  to  subjects  so  that  adequate  levels  of  performance  (range 
intervals)  are  maintained,  as  characterized  by  the  C2  Approach  they  should  be  operating  at  (e.g.,  allocation  of  decision 
rights  and  setting  goals  at  team  and  organization  level). 


Details  about  manipulation  of  independent  variables  are  provided  in  Table  3. 

The  model  also  includes  the  following  variables  of  interest  (intermediate  dependent  variables)4: 
Quality  of  Individual  and  Shared  Information; 

Quality  of  Individual  and  Shared  Awareness  and  Understanding; 

Quality  of  Individual  and  Shared  Decisions;  and 
Self-Synchronization. 


The  model  dependent  variable  of  interest  is  Task  Performance ,  which  measures  individual,  team  and  collective 
effectiveness  and  efficiency  (given  effectiveness). 


4  These  variables  are  described  in  depth  in  available  CCRP  literature  (e.g.,  (Alberts  et.  al.  2001)  and  (Alberts  and  Hayes  2006))  and  NATO  Publications  (SAS-050 
2006).  The  mapping  between  C2  variables  and  ELICIT  was  made  in  (Manso  and  Nunes  2008). 
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Moreover,  relevant  additional  intervening  variables,  labeled  as  enablers/inhibitors,  were  identified  with  influence  on 
the  model.  Their  manipulation  is  out  of  the  scope  of  this  work,  except  for  Task  Difficulty.  These  variables  are 
described  in  Table  16  in  Annex. 


Name 

Description 

Network  Characteristics  and 
Performance 

Capabilities  of  the  network  as  defined  in  terms  of  reach , 
availability ,  bandwidth  and  responsiveness  (SAS-050  2006). 

Reach  is  controlled  by  allowing  (or  prohibiting)  network 
interactions  between  subjects  and  teams.  All  others  (< availability , 
bandwidth  and  responsiveness)  will  be  fixed  for  all  experiments 
(ELICIT  platform  performance  provides  good  availability, 
bandwidth  and  responsiveness). 

This  variable  affects  PI-C  and  DI-C. 

Information  Sharing  and  (incentives 
for)  Collaboration 

This  variable  may  be  controlled  and  influenced: 

-  Control:  distribution  of  information  by  the  ELICIT  platform. 

All  factoids  are  distributed  and  made  available  to  subjects  in  three 
waves. 

-  Influence:  distribution  of  information  as  a  result  of  human 
sharing  and  posting. 

Regardless  of  the  infra/info-structure  capabilities  provided,  it  will 
be  the  subjects’  will  that  will  determine  the  extent  of  collaboration 
(how  and  if).  We  will  attempt  to  induce  /  influence  collaborative 
behavior  by: 

-  defining  collective  or  isolated  goals 

-  set  individual  and  collective  decision  rights  (see  ADR) 

See  also  Individual  and  Team  Characteristics  (in  Annex). 

This  variable  affects  PI-C  and  DI-C. 

Allocation  of  Decision  Rights 

Decision  rights  will  be  allocated  according  to  the  C2  Approach  to 
implement5: 

-  Distributed  for  higher  maturity  approaches; 

-  None  /  (de)centralized  for  lower  maturity  approaches. 

This  variable  is  a  C2  dimension. 

Table  3  -  Model  independent  variables 


EXPERIMENT  DESIGN 

Using  the  ELICIT  platform  to  observe  various  approaches  to  command  and  control  that  correspond  to  different  level 
of  C2  maturity  described  in  the  N2C2M2  requires  modifying  the  ELICIT  baseline  organizational  specifications 
(Ruddy  2007)  (Ruddy  2008)  as  well  as  making  modifications  to  the  environment  itself. 

For  the  N2C2M2  experimentation,  the  scenario  (terrorist  attack)  and  nature  of  the  problem  is  kept  (i.e.,  determine 
who ,  what ,  where  and  when),  as  well  as  the  factoids  sets6,  organization  size  (17  subjects)  and,  when  applicable, 
organization  structure  (i.e.,  three  levels:  overall  coordinator,  4  teams  each  with  1  team  leader  and  3  team  members). 
The  information  sharing  capabilities  are  also  maintained  (i.e.,  share,  post  and  pull  actions)  but,  for  higher  maturity 
levels,  factoids  assessment  is  allowed  as  a  way  to  enhance  knowledge  sharing  between  subjects.  Moreover, 
interactions  allowed  between  teams  and  subjects  is  now  fully  controlled  as  part  of  the  ELICIT  setup. 

Table  4  provides  an  overall  view  of  what  is  being  manipulated  in  ELICIT  to  represent  each  of  the  N2C2M2  Collective 
C2  approaches.  A  detailed  description  of  the  experimental  design  is  presented  in  “Detailed  Design  of  Experiments”  in 
Annex. 


5  Followed  a  similar  approach  adopted  by  (Chong  et.  al.  2007). 

6  Factoids  sets  were  reused  for  these  experiments,  but  were  translated  to  Portuguese  language.  Moreover,  reclassification  in  relevance  was  performed  (see 
“Solution  Logic”  in  Annex). 
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Independent 

Variables 

Collective  C2  n. 
Approach 

Network  Characteristics  and 
Performance 

(Incentives  for) 

Information  Sharing7  and 
Collaboration 
(Initial  conditions) 

Allocation  of  Decision 
Rights  to  the  Collective 

Organizational 

Structure 

Model  5: 

Edge 

Fully  connected  and 
interoperable. 

Existing  P2P  connectivity 
between  all  individuals. 

Shared  team  websites. 

Collective  goals  set  to  all  individuals  (no 
predefined  roles)  and  unrestricted 
communications  policies  (across 
individuals  with  shared  websites  and 
factoids  evaluation)  should  increase  and 
enrich  interactions  and  collaboration 
among  all  individuals. 

Fully  distributed  /  not  explicit 
(per  individual)  and  dynamic. 
Individuals  choose  which  part 
(or  parts)  of  the  problem 
space  they  work. 

1  team  with  17  team 
members  (non- 
hierarchical). 

Model  4: 
Collaborative 

Fully  connected  and 
interoperable. 

Existing  P2P  connectivity 
between  all  individuals. 

Shared  team  websites. 

Collective  goals  set  to  Coordinator  and 
Team  leaders  together  with  unrestricted 
communications  policy  (across 
individuals  with  shared  websites  and 
factoids  evaluation)  should  increase  and 
enrich  interactions  and  collaboration 
among  individuals  (still  with  stronger 
ties  expected  between  Coordinator  and 
Team  leaders). 

Distributed  and  collaborative. 
Across  Team  leaders  and 
coordinator/ facilitator. 
Coordinator/facilitator  works 
in  all  problems. 

4  teams  (with  1  team 
leader)  and  1 
Coordinator/F  acilitator. 

Model  3: 

Coordinated 

Minimum  connectivity. 
Stove-pipe:  between  Team 
leaders  and  coordinator. 

Teams  exclusive  access  to 
their  websites. 

Coordinator  access  to  all 
websites. 

Collective  goals  centralized  by  function 
(Coordinator,  assisted  by  Team  leaders) 
should  enable  stronger  interactions 
among  hierarchies  and  subordinates 
(Coordinator  and  Team  leaders  and 

Team  leaders  and  Team  members). 

Centralized  (in  Coordinator). 
Team  specialized  problem 
space. 

4  teams  (with  1  team 
leader)  and  1 
Coordinator/F  acilitator. 

Model  2: 

De-conflicted 

Minimum  connectivity. 
Stove-pipe:  between  Team 
leaders  and  Information 

Broker. 

Teams  exclusive  access  to 
their  websites. 

Isolated  goals,  but  factoids 
interdependency  should  enable 
weak/minimum  interactions,  between 
stove-pipes  (Deconflictor  and  Team 
leaders). 

Established  constraints  (share 
what  is  relevant  to  other 
teams). 

Decision  allocated  to  each 
Team  leader. 

Team  specialized  problem 
space. 

4  teams  (with  1  team 
leader)  and  1 
information  broker. 

Model  1:  Conflicted 

Teams  exclusive  access  to 
their  website. 

Non-interoperable  (no  cross¬ 
teams  communications). 

None  outside  teams. 

None 

(independent  decision  rights 
within  teams  only) 

4  isolated  teams  a  1 
isolated  subject. 

Table  4  -  Five  Organizational  Models  for  Five  C2  Approaches:  in  a  Nutshell 


Part  of  the  incentive  policies  implemented  is  based  on  setting  organization  goals  according  to  adopted  approach,  as 
presented  in  Table  5. 


Collective  C2  Approach 

Criteria  for  Mission  Effectiveness 

Model  5:  Edge 

Set  at  Organization  level  (fully-distributed). 

Each  member  may  work  on  any  problem  space  and  the  most  frequent 
ID  (mode  value)  in  each  problem  space  must  be  correct. 

Model  4:  Collaborative 

Set  at  Organization  level  (Coordinator  and  Team  Leaders). 

Coordinator  must  determine  the  correct  solution  in  all  problem  spaces 
OR  Team  Leader  (assigned  a  problem  space)  must  determine  the 
correct  solution  for  his  problem  space. 

Model  3:  Coordinated 

Set  at  Organization  level  (centralized  at  Coordinator). 

Coordinator  must  determine  the  correct  solution  in  all  problem 
spaces. 

Model  2:  De-conflicted 

Set  at  Team  Level  only. 

Each  Team  Leader  is  assigned  a  problem  space  and  he  must 
determine  the  correct  solution  to  his  problem  space. 

Model  1 :  Conflicted 

Set  at  Team  Level  only. 

Each  Team  Leader  is  assigned  a  problem  space  and  he  must 
determine  the  correct  solution  to  his  problem  space. 

Table  5  -Mission  Effectiveness  criteria  per  C2  Approach 


The  ELICIT  platform  distributes  all  factoids  in  three  waves. 
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EXPERIMENTATION  BASELINE 

The  experimentation  baseline  comprises  collected  data  from  runs,  which  includes  ELICIT  setup  files,  preparation 
material  (Subject’s  instructions  and  ELICIT  setup  files),  subjects’  questionnaires  and  ELICIT  logs  (see  64 
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Data  Collection  and  Analysis  Process”  in  Annex).  The  list  of  runs  analyzed  is  presented  in  Table  6. 


ID 

Date 

Log  File 

C2  Approach 

Factoid 

Set 

LI -01 

13-05-2009 

20090513-1 053-21 81 2-group_1_CONFLICTED.log 

CONFLICTED 

4 

LI -02 

28-05-2009 

20090528-1246-32783-group_1_CONFLICTED.log 

CONFLICTED 

1 

LI -03 

03-06-2009 

20090603-1135-40719-group_1_CONFLICTED.log 

CONFLICTED 

3 

L2-01 

29-04-2009 

20090429-1308-24386-group_1_B.log 

DECONFLICTED 

1 

L2-02 

29-04-2009 

20090429-1402-18407-group_1_B.log 

DECONFLICTED 

4 

L2-03 

12-05-2009 

20090512-1511-12023-group_1_DECONFLICTED.log 

DECONFLICTED 

1 

L2-04 

02-06-2009 

20090602-1413-19588-group_1_DECONFLICTED.log 

DECONFLICTED 

3 

L3-01 

07-05-2009 

20090507-1413-26985-group_1_COORDINATED.log 

COORDINATED 

1 

L3-02 

13-05-2009 

20090513-1142-59475-group_1_COORDINATED.log 

COORDINATED 

3 

L3-03 

03-06-2009 

20090603-1212-12301-group_1_COORDINATED.log 

COORDINATED 

2 

L3-04 

03-06-2009 

20090603-1332-06152-group_1_COORDINATED.log 

COORDINATED 

4 

L4-01 

06-05-2009 

20090506-1133-24903-group_1_COLLABORATIVE.log 

COLLABORATIVE 

1 

L4-02 

06-05-2009 

20090506-1225-19696-group_1_COLLABORATIVE.log 

COLLABORATIVE 

4 

L4-03 

12-05-2009 

2009051 2-1 41 5-1 91 71  -group_1_COLLABORATIVE.log 

COLLABORATIVE 

3 

L4-04 

02-06-2009 

20090602-1515-03801-group_1_COLLABORATIVE.log 

COLLABORATIVE 

2 

L5-01 

29-04-2009 

20090429-1101-06528-group_1_A.log 

EDGE 

1 

L5-02 

29-04-2009 

20090429-1152-08894-group_1_A.log 

EDGE 

4 

L5-03 

07-05-2009 

20090507-1500-37261-group_1_EDGE.log 

EDGE 

4 

Table  6  -  N2C2M2  experiment  runs 


About  4  valid  runs  were  conducted  per  C2  approach,  covering  all  factoids  sets  available8.  However,  the  following 
faults  occurred:  for  level  2,  factoid  set  1  was  used  twice  and  factoid  set  2  was  not  used;  for  level  5,  only  3  valid  runs9 
were  conducted,  factoids  set  4  was  used  twice  and  factoids  set  2  and  3  were  not  used. 

Collected  data  allowed  identifying  differentiating  aspects  between  each  C2  approach,  but  we  are  aware  that  more  data 
(hence,  more  runs)  are  necessary  to  conduct  a  robust  statistical  analysis  and  to  consolidate  findings. 

The  experiments  analysis  is  presented  in  next  section. 

EXPERIMENTS  ANALYSIS 

For  the  experiments  analysis,  several  C2  variables  are  measured  so  that  approaches  may  be  objectively  characterized 
and  compared.  In  essence,  we  intend  to  identify  quantitative  and  qualitative  aspects  differentiating  the  C2  approaches. 

The  following  is  measured: 

•  Information  Domain:  level  of  information  accessible,  level  of  overall  shared  information  reach  and  level 
of  information  reached  for  key  roles  (CTC  and  team  leaders)10.  Finally,  awards  are  given  based  on 
information  scores  with  timestamp  (indicating  when  score  was  achieved). 

•  Interactions  and  Social  Domain:  nature,  quantity  and  quality  of  the  interactions  (number  of  shares,  posts 
and  pulls  actions  per  subject,  team  and  overall  organization),  network  reach  and  deviation  of  inward  and 
outward  flows  from  the  mean. 

•  Cognitive  Domain:  correct  understanding  (extent  of)  for  overall  and  each  problem  space  and  a  measure  of 
self-synchronization  (labeled  as  Cognitive  Self-Synchronization).  Awards  are  given  based  on  extent  of 
understanding  scores  with  timestamp. 

•  Measures  of  Merit:  effectiveness  and  efficiency  (in  time  and  cost). 


8  Trial  runs,  which  preceded  runs  for  analysis,  were  not  included  in  table. 

9  An  EDGE  run  was  invalidated  because  recorded  logging  contained  inconsistencies  between  setup  and  actual  organization  approach. 

10  It  should  be  noted  that: 

•  Overall  shared  information  reach  is  the  set  of  information  which  may  be  reached  by  all  subjects,  and 

•  Information  reached  is  the  set  of  information  which  has  been  reached  by  a  subject  (i.e.,  information  is  considered  accessed  when  a  share  is  received  or  a 
pull  is  performed). 
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Data  measurements  will  be  presented  for  all  runs  and/or  approaches.  Presentation  of  mean  values  will  include 
reference  to  maximum  and  minimum  values11.  Most  tables  and  charts  used  should  be  self-explainable,  except  for 
statistical  charts  and  mean  deviation  chart  which  are  briefly  explained  in  Annex. 

Information  Domain 

All  new  factoids  are  distributed  by  server  in  3  waves12  (see  Table  3,  Information  Sharing  row).  By  default,  waves  1,  2 
and  3  occur  at  0,  5  and  10  minutes  respectively.  However,  due  to  a  misconfiguration,  some  runs  used  the  following 
setup:  wave  1  @  0  minutes,  wave  2  @  1  minute  and  wave  3  @  2  minutes.  Affected  runs  were:  Ll-03,  L3-03,  L3-04 
and  L4-04.  This  modification  will  be  considered  when  analyzing  the  experiments. 

All  information  (68  factoids)  becomes  accessible  after  wave  3.  Figure  3  presents  increase  in  information 
accessibility  across  time. 


Information  Accessible  (Serv 

70  _ 

65 


I  60 

%  55 
u-  50 


Time  (sec) 


|»  Accessibility  Index  (Relevant)  •  Accessibility  Index  (All)  AAccessil: 


Figure  3  -  Information  Accessibility  across  time:  default  (left)  and  changed  (right)  modifications 


The  server  distribution  of  factoids  to  subject  is  further  depicted  in  Figure  4.  It  is  clear  that  factoids  are  uniquely 


distributed  to  subjects.  Note  that  colors  represent  factoids  relevance  (Green=K/E,  Orange=S,  Grey=N). 


Figure  4  -  Server  Distribution  of  Factoids  to  Subjects 


As  a  result  of  share  and  post  actions,  factoids  become  accessible  to  more  subjects  and,  therefore,  individual  and  shared 


Figure  5  -  Factoids  Distribution  to  Subjects  after  shares  and  posts  actions  (Edge  run  L5-02) 


11  Standard  deviation  will  not  be  used  since  few  data  is  available. 

12  Each  factoid  set  has  64  factoids.  For  all  experiments,  platform  is  setup  to  disseminate  factoids  as  follows:  wave  1  sends  32  factoids,  wave  2  sends  17  factoids 
and  wave  3  sends  17  factoids  (all  factoids  are  made  available  to  subjects  after  wave  3).  Each  wave  distributes  about  the  same  proportion  of  relevant  and  noise 
factoids.  Platform  is  setup  so  that  factoids  are  first  disseminated  to  only  one  subject  (i.e.,  sharing  of  information  between  any  two  subjects  as  a  result  of  factoids 
server  distribution  is  always  zero).  See  also  (Ruddy  2007). 
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The  amount,  extent  and  speed  of  information  shared  are  strongly  influenced  by  the  organizational  approach  used. 
Figure  6  presents  scores13  achieved  in  key  variables  of  the  information  domain  across  C2  approaches. 


Shared  Information  Reach  was  measured  as  the  set  of  information  (i.e.,  factoids)  which  is  accessible  to  all  subjects. 
Only  COLLABORATIVE  and  EDGE  approaches  obtained  values  other  than  zero:  COLLABORATIVE  obtained  an 
average  value  of  65%  (ranging  between  50%  and  80%)  while  EDGE  obtained  the  best  overall  score  with  an  average 
value  of  75%  (ranging  between  65%  and  80%).  Both  organizations  were  very  effective  sharing  information,  but  none 
reached  100%,  meaning  that  local  hoarding  of  factoids  occurred. 

Given  zero  scores  for  CONFLICTED,  DECONFLITED  and  COORDINATED,  individual  measurements  are  therefore 
necessary  to  assess  performance  in  distribution  of  information  for  these  approaches  (COLLABORATIVE  is  also 
included  for  completeness,  while  role  analysis  is  not  applicable  for  EDGE).  For  these  approaches,  measurements  will 
focus  on  key  roles,  namely,  the  overall  coordinator  (CTC,  a.k.a.  ALEX)14  and  team  leaders. 

Relevant  Information  Reached  was  measured  as  percentage  of  relevant  information  (i.e.,  K,  E  or  S  factoids)  actually 
accessed  (received  after  a  share  or  after  pull)  by  a  subject.  Value  for  CONFLICTED  was  zero  (since  CTC  is  isolated, 
he  can  only  reach  the  4  factoids  sent  by  server),  then  increased  to  35%  (min:  30%,  max:  50%)  in  DECONFLICTED, 
COORDINATED  had  maximum  overall  score  with  76%  (min:  65%,  max:  80%)  and  COLLABORATIVE  had  a 
similar  score  with  73%  (min:  65%,  max:  80%).  In  fact,  the  result  obtained  by  the  COORDINATED  approach  is 
interesting  for,  although  overall  shared  information  reach  score  is  zero,  the  organization  was  very  effective  in  sharing 
information  to  their  centralized  decision  maker.  Note  also  that  CTC  is  the  main  responsible  for  zero  score  in 
information  sharing  in  DE-CONFLICTED  and  COORDINATED  (see  organizations’  setup  in  Annex),  meaning  that 
subjects  playing  this  role  behaved  as  information  aggregators  and  hoarders. 

Average  (value  of)  Relevant  Information  Reached  by  Team  Leaders  was  also  measured  to  evaluate  each  of  the 
approaches’  ability  to  disseminate  information  across  teams  and  to  key  roles.  The  following  can  be  concluded: 

•  Except  for  CONFLICTED  (EDGE  being  not  applicable),  results  obtained  in  all  other  approaches  were  always 
inferior  to  those  obtained  for  the  CTC.  This  indicates  that  subjects  having  the  overall  coordinator  role  usually 
(i)  hoard  some  information  from  team  leaders  or  (ii)  team  leaders  don’t  reach  information  which  was  made 
available  to  them.  The  highest  difference  in  value  was  observed  for  COORDINATED  approach  in  which 
team  leaders  reached  an  average  value  of  29%  (min:  23%,  max:  40%)  while  CTC  reached  an  average  of  76%. 

•  Only  COLLABORATIVE  achieved  above  average  scores  (>50%),  while  both  DECONFLICTED  and 
COORDINATED  remained  below  30%  (CONFLICTED  was  limited  to  about  10%).  All  these  approaches 
failed  in  making  information  available  across  teams. 


13  Scores  were  awarded  when  values  reached  10%,  20%,  30%,  50%,  65%,  80%  and  100%  of  total. 

14  More  precisely,  isolated  coordinator  for  CONFLICTED,  information-broker  for  DECONFLICTED,  cross-team  coordinator  (CTC)  for  COORDINATED  and 
CTC/facilitator  for  COLLABORATIVE.  This  role  is  always  played  by  subject  named  by  ELICIT  as  ALEX.  For  simplicity,  this  role  will  be  referred  as  CTC. 
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Illustrative  examples  are  provided  in  ANNEX  ‘Information  Reach  and  Information  Reached  by  Subjects’.  The 
following  is  outlined: 

•  Both  EDGE  and  COLLABORATIVE  were  very  efficient  disseminating  information  across  the  organization, 
but  local  hoarding  always  existed.  Table  19  and  Table  20  (top  part)  illustrate  two  EDGE  runs  were  most 
information  is  shared:  the  former  included  hoarding  of  3  K/E  facts  (and  few  S  and  N)  while  the  later 
succeeded  in  having  all  K/E  within  reach  of  all  Subjects.  Likewise,  two  COLLABORATIVE  runs  are 
illustrated  in  Table  21  and  Table  22:  in  the  former  a  few  factoids  are  hoarded  (including  K/E)  while  in  the 
later  more  critical  information  was  shared  across  organization.  Yet,  observing  results  for  information  reached 
provides  further  interesting  insight  on  subjects’  behavior:  although  most  subjects  were  information  seekers 
(see  “Interactions  and  Social  Domain”  section),  a  few  didn’t  reach  all  accessible  information  (as  a  result  of  no 
shares  or  no  sufficient  pull  actions  on  websites).  This  is  illustrated  in  bottom  part  of  Table  21  and  Table  22. 
Illustrative  and  extreme  examples  for  EDGE  and  COLLABORATIVE  are  provided  in  Table  20  (bottom  part) 
by  Jesse  and  Table  22  (bottom  part)  by  Harlan  (Table  17  presents  Information  Domain  Score  detailed  data). 

•  COORDINATED  approaches  were  very  efficient  disseminating  information  to  CTC,  but  not  to  overall 
organization.  Table  23  illustrates  a  run  where  almost  all  key  factoids  were  effectively  made  available  to 
ALEX,  except  55K,  which  provided  critical  information  about  WHEN-day  (interesting,  this  factoid  was 
shared  between  WHERE  team  members  but  not  to  the  WHERE  team  leader).  Moreover,  information  was 
heavily  partitioned  to  teams.  Table  23  also  illustrates  a  particular  case  where  there  is  no  shared  information 
between  any  two  teams. 

•  DECONFLICTED  approaches  had  a  decrease  in  shared  information  for  Information  Broker  (ALEX)  and 
Team  Leaders.  As  illustrated  by  Table  24,  it  is  interesting  to  note  the  impact  of  changing  roles  on  subjects’ 
sharing  behavior  (this  aspect  is  further  analyzed  in  “Interactions  and  Social  Domain”  section):  Team  Leaders 
had  the  main  role  in  this  organization:  the  extent  of  information  shared  to  ALEX  was  less  than  in 
COORDINATED  (some  information  was  retained  within  teams) 

•  CONFLITED  is  presented  here  for  completeness.  Table  25  illustrates  partition  of  information  across  teams 
and  isolated  coordinator,  which  couldn’t  be  shared  across  teams  due  to  setup  connectivity  restrictions. 

An  essential  aspect  was  to  measure  the 
ability  to  share  critical  information  to 
key  positions  (the  right  information  in 
the  right  position).  Critical  information 
was  considered  as  the  minimum  subset 
of  factoids  required  to  determine  a 
problem  space  (see  “Solution  Logic”  in 
Annex),  according  to  following 
criterion: 

•  CTC  needs  critical  information  for 
all  problem  spaces 

•  TL  and  TMs  needs  critical 
information  for  their  problem  space. 

•  In  EDGE,  all  TMs  need  critical 
information  for  all  problem  spaces. 

Variable  measured  was  Percentage  of  Critical  Information  Missing  (per  approach  and  per  role)  presented  in  Figure 
7.  As  maturity  increases,  percentage  of  critical  information  missing  in  key-roles  decreases.  EDGE  reaches  the  lowest 
score  with  a  low  variation  between  extremes,  COLLABORATIVE  succeeded  making  critical  information  across  roles 
(although  variation  increased  as  role  position  decreased).  The  change  between  COORDINATED  and 
DECONFLICTED  approaches  illustrates  the  heavy  asymmetry  between  CTC  and  TLs  already  mentioned  here.  The 
scale  is,  nevertheless,  impressive:  CTC  decreased  from  10%  to  55%  information  missing,  while  TLs  variation 
remained  relatively  stable  (with  an  increase  between  min-max  variation  from  L3  to  L2).  As  expected,  CONFLICTED 
had  the  highest  percentage  of  critical  information  missing. 

An  assessment  on  the  approaches  performance  in  the  information  domain,  according  to  information  scores  and  time 
efficiency,  yield  the  results  detailed  in  Table  7. 


Critical  Information  Missing  (percentage) 


CTC  |  TLs  |  TMs 
CONFLICTED 


CTC  |  TLs  |  TMs 
DECONFLICTED 


CTC  |  TLs  |  TMs 
COORDINATED 


CTC  |  TLs  |  TMs 
COLLABORATIVE 


Figure  7  -  Percentage  of  Critical  Information  Missing 


-  Page  14  of  59  - 


15th  ICCRTS:  The  Evolution  ofC2 

N2C2M2  Experimentation  and  Validation  :  Understanding  Its  C2  Approaches  and  Implications 


Information  Domain: 

LI  to  L3  RELEVANT  Information  Reached  (CTC) 

L4  to  L5:  Shared  Information  (accessible)  (OVERALL) 

Run  ID 

LEVEL 

10% 

20% 

30% 

40% 

65% 

80% 

100% 

LI-01 

LI 

LI -02 

LI 

LI -03 

LI 

L2-01 

L2 

313 

621 

1110 

L2-02 

L2 

143 

340 

582 

1057 

L2-03 

L2 

110 

387 

757 

L2-04* 

L2 

106 

396 

748 

1748 

L3-01 

L3 

42 

98 

265 

420 

665 

1805 

L3-02 

L3 

19 

37 

80 

324 

600 

680 

L3-03* 

L3 

36 

91 

113 

202 

304 

1420 

L3-04* 

L3 

57 

107 

162 

390 

878 

1432 

L4-01 

L4 

38 

112 

329 

514| 

1351 

L4-02 

L4 

27 

39 

58 

321 

523 

702 

L4-03 

L4 

123 

293 

338 

622 

L4-04* 

L4 

43 

78 

113 

162 

234 

L5-01 

L5 

64 

95 

320 

627| 

2219 

L5-02 

L5 

38 

62 

97 

360 

659 

1304 

L5-03 

L5 

94 

142 

208 

530 

652 

1172 

*  Early  delivery  of  factoids  from  server  as  illustrated  in  Figure  3 
Table  7  -  Information  Domain  Scores 


For  shared  information  accessible,  the  best  overall  scores 
were  obtained  by  EDGE  approaches.  2  out  of  3  runs  reach  a 
score  of  80%  (at  1172  and  1304  seconds).  However, 
COLLABORATIVE  had  the  best  efficiency:  80%@702 
seconds.  2  out  of  4  runs  obtained  65%,  one  reached  very 
early  (234  seconds),  as  a  consequence  of  subjects’  shares 
and  posts  and  early  distribution  of  factoids  (see  Figure  3 
right  part). 

Since  the  other  three  approaches  didn’t  achieve  minimum 
score  of  10%  for  shared  information  accessible,  their 
assessment  individual  scores  were  awarded  based  on  the 
percentage  of  the  relevant  information  reached  by  CTC15. 
All  4  COORDINATED  runs  achieved  a  score  of  80%,  of 
which  L3-02  had,  exceptionally,  a  good  efficiency  (680 
seconds).  Moreover,  in  this  approach,  Team  Leaders 
usually  achieve  a  low  score  of  20%  (see  Table  18  in 
Annex). 

DECONFLICTED  runs  achieved  40%  and  30%  scores  for 
CTC,  demonstrating  that  the  amount  of  information  kept 
within  each  team  reach  was  substantial.  Again,  the  data 
provided  by  Table  18  (in  Annex)  displays  a  slight 
improvement  in  information  accessed  by  Team  Leaders, 
which,  at  most,  reached  50%. 

CONFLICTED  didn’t  reach  minimum  scores  for  ALEX 
while  Team  Leaders  reached  between  10%  and  20%  of  the 
information  (see  Table  18). 

Finally,  the  change  in  the  periodic  delivery  of  server 
factoids  had  not  a  significant  effect  (see  L2-04,  L3-03,  L3- 
04),  except  for  L4-04  run  (took  less  time  to  reach  a  high 
score). 


Interactions  and  Social  Domain 

Interactions  in  ELICIT  consist  of  all  subjects’  shares,  posts  and  pulls  actions.  Depending  on  the  approach,  their 
number,  frequency,  distribution  and  reach  may  change. 

We  start  by  analyzing  the  number  of  actions  per  hour  (since  duration  of  runs  is  not  exactly  the  same).  The  results  are 
presented  in  Figure  8  (detailed  data  is  presented  in  Table  26  and  Table  27). 


15  Assessment  of  CTC  scores  intended  to  evaluate  his  pro-activity  to  seek  information  (hence  REACHED)  and  quality  of  data  achieved  (hence  RELEVANT). 
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There  are  clear  differences  between  the 
five  approaches. 

•  Number  of  pulls  per  hour  reached  its 
highest  values  for  EDGE  approaches 
(maximum  value  was  1443  for  L5-01). 
EDGE  Subjects  were  information 
seekers.  COLLABORATIVE  approach 
followed  with  second  highest  values 
(about  half  of  EDGE).  All  other 
approaches  had  few  pulls  per  hour 
(ranging  from  109  to  242). 

•  Number  of  posts  per  hour  reached  its 
highest  value  for  COLLABORATIVE 
approaches  (maximum  value  was  372 
for  L4-04),  followed  by  all  other 
approaches  with  no  significant 
differences. 

•  Number  of  shares  per  hour  reached  its 
highest  value  for  DECONFLICTED 
approach  (maximum  value  was  455  for 
L2-04),  followed  by  COORDINATED. 
These  two  approaches  required  sharing 
to  pass  information  across  teams,  but 
DECONFLICTED  clearly  performed 
more  shares  (from  CTC  to  TLs).  Both 
EDGE  and  COLLABORATIVE  had 
the  lowest  sharing  activity. 

•  Number  of  IDs  per  hour  reached  its 
highest  value  for  EDGE,  followed  by 
COLLABORATIVE. 

Both  COLLABORATIVE  and,  especially,  EDGE  had  intensive  pull  activity  (see  Figure  8,  green  bar)  representing 
66%  and  82%  (of  share,  post  and  pull)  respectively.  Key-factors  may  have  been  the  availability  of  all  websites  to  all 
subjects  and  the  instructions  to  induce  collaboration  and  sharing.  Moreover,  EDGE  subjects  had  yet  another  factor 
enabling  individual  pro-activity  towards  information  seeking  which  was  distribution  of  decision  rights  (i.e.,  each 
subject  identification  was  relevant  to  determine  organization  effectiveness). 

DECONFLICTED  and  COORDINATED  approaches  required  share  actions  to  share  information  across  teams. 
Therefore,  it  is  not  surprising  that  shares  activity  was  higher  than  those  observed  for  COLLABORATIVE  or  EDGE. 
Another  interesting  aspect  was  to  observe  a  decrease  in  sharing  activity  between  DECONFLICTED  and 
COORDINATED,  likely  to  be  a  consequence  of  changing  role  from  Information  Broker  (share  intensive  role)  to  a 
centralized  approach  (propensity  to  receive  shares  from  team  leaders  but  not  send  shares  to  team  leaders). 

The  aspect  of  role  influence  in  subjects’  behavior  and  related  asymmetries  may  be  better  understood  by  measuring 
subjects  deviation  in  terms  of  outflows  (OUT:  shares  sent  and  posts,  i.e.,  distribute  information)  and  inflows  (IN: 
shares  received  and  pulls,  i.e.,  receive  and/or  seek  information)  from  normal  (or  average)  behavior. 

That  is,  for  a  given  subject  5) : 

IN  DEV s  =  (nbr  _ shares  _received s  +  nbr  _ pulls s  )  -  {Average _nbr  _ shares  _received  +  Average _nbr  _ pulls) 
OUT _DEVS  =  ( nbr  _ shares _ sents  +  nbr  _ posts s  )  -  {Average _ nbr  _ shares _ sent  +  Average _nbr  _ posts ) 

Deviation  in  terms  of  IN  and  OUT  behavior  is  expressed  in  absolute  terms,  allowing  to  interpret  its  true  proportions. 


■  Shares  per  Hour 

■  Posts  per  Hour 

■  Pulls  per  Hour 

■  Ids  per  Hour 


Ll-01  Ll-02  Ll-03  L2-01  L2-02  L2-03  L2-04  L3-01  L3-02  L3-03  L3-04  L4-01  L4-02  L4-03  L4-04  L5-01  L5-02  L5-03 


Nature  of  Interactions  (distribution) 

100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


Figure  8  -  Nature  of  Interactions:  totals  per  hour  (top)  and  distribution  (bottom)  per 
approach 


CONFLICTED  DECONFLICTED  COORDINATED  COLLABORATIVE  EDGE 

■  Shares  per  Hour  ■  Posts  per  Hour  Pulls  per  Hour 
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The  average  values  and  standard  deviations  per  role  (i.e.,  CTC,  TL  and  TM)  and  per  approach  were  calculated16. 
Figure  9  illustrates  the  resulting  values,  discriminated  per  approach  and  per  role.  The  chart  needs  explanation  for 
proper  interpretation,  provided  in  “Explanation  of  IN  and  OUT  Flows  Chart”  in  Annex. 

IN  propensity:  all  coordinator  and  leader  roles 
displayed  a  higher  deviation  for  IN  propensity 
(all  red  circumferences  are  located  on  the  right 
size).  Highest  value  occurred  for 
DECONFLICTED,  followed  by 

COORDINATED.  Leader  roles’  IN  propensity 
in  CONFLICTED  and  COLLABORATIVE 
were  positive  but  close  to  average.  Team 
members  always  had  a  negative  deviation  for 
IN  propensity  in  all  four  approaches. 

OUT  propensity:  Leader  role  in 

DECONFLICTED  showed  a  significant  OUT 
positive  deviation,  followed  by 
COORDINATED:  such  is  consistent  with 
organizational  need  for  leaders  to  disseminate 
information  across  teams.  Team  member  roles 
were  close  to  average,  except  in  DECONFLICTED,  which  was  below  average. 

Figure  27  (last  column)  presents  node  balance  (IN  and  OUT  activity)  per  approach  for  a  particular  run.  It  is 
interesting  to  note  the  following  particular  cases: 

•  CTC  had  a  major  deviation  from  L2  to  L3:  significant  decrease  in  OUT  and  significant  increase  in  IN  given 
its  change  from  Information  Broker  (sharer)  to  Coordinator  (information  aggregator).  TLs  also  had  a 
significant  deviation  from  L2  to  L3:  decrease  in  IN  and  OUT. 

•  TMs  IN  and  OUT  activity  in  L2  and  L3  had  a  low  deviation  from  normal  behavior  (small  role  relevance). 
Deviation  increased  for  L4  and  L5  (higher  role  relevance).  TMs  may  have  perceived  an  increase  in  relevance 
as  maturity  with  increased  maturity. 

Noted  asymmetries  are  consistent  with  expectations:  excluding  CONFLICTED  (which  only  considers  interactions 
within  teams),  on  average,  behavioral  asymmetries  between  leader  roles  and  member  roles  decrease  as  the  maturity 
approach  increases. 

Finally,  subjects’  IN  and  OUT  natural  propensity  should  not  be  ignored  when  dealing  with  role  assignment.  The 
values  obtained  for  standard  deviation  (see  Table  29)  indicate  a  high  variation  from  the  mean.  Therefore,  although 
general  patterns  were  noted,  subjects  behaved  differently  at  individual  level  (perhaps  influenced  by  their  own 
personality  and  social  traits)  and,  therefore,  not  always  as  adequately  as  the  job  required. 

Another  relevant  measure  consists  of  the  network  reach,  measured  as  the  percentage  of  the  network  that  each  node  (or 
subject)  reached  or  was  reached  by,  i.e.,  sent  to  or  received  from  share  actions.  The  following  variables  were 
measured:  maximum,  average  and  minimum  (per  run  and  per  approach).  Peer-to-peer  (p2p)  indicators  for  network 
inclusion  and  network  exclusion  were  obtained  by  counting  the  number  of  nodes  with  more  than  50%  (7  or  more 
subjects  reached  which  means  high  network  inclusion)  and  less  than  10%  (less  than  1  subject  reached  meaning  high 
network  exclusion)  network  reach.  This  analysis  only  considered  direct  peer-to-peer  actions  (i.e.,  share)  and  didn’t 
consider  any  post  activity.  Results  are  presented  in  Table  8  and  overall  chart  is  presented  in  Figure  10. 


IN  and  OUT  Flows  (deviation  from  average):  per  role  and  per  approach 


i,0  -1! 

5,0  -5,0 

,o\  is 

,0  25 

O  CONFLICTED  (TLs) 

•  CONFLICTED  (TMs) 

O  DECONFLICTED  (IB/TLs) 

•  DECONFLICTED  (TMs) 

,o  O  COORDINATED  (CTC/TLs) 

O  COORDINATED  (TMs) 

O  COLLABORATIVE  (CTC/TLs) 
O  COLLABORATIVE  (TMs) 


Figure  9  -  IN  and  OUT  Flows  per  role  and  per  approach 


16  The  calculations  kept  their  original  units  so  that  true  proportions  can  be  assessed  across  different  approaches.  The  detailed  values,  per  run,  are  presented  in  Table 
29  in  Annex. 
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Figure  10  -  P2P  Network  Reach 


EDGE  approach  obtained 
the  highest  network  P2P 
reach  (71%  for  L5-03) 
and  maximum  average 
value  (57%),  followed  by 
COLLABORATIVE. 

Both  EDGE  and 
COLLABORATIVE  also 
obtained  highest  network 
exclusion.  For  example,  it 
is  surprising  to  note  that 
EDGE  run  L5-03  had  14 
nodes  that  reached  less 
than  10%  of  the  network. 

Network  reach  for  levels 

3  and  below  were  limited  by  ELICIT  setup,  but  average  values  were  closer  to 
maximum  allowed  (24%  for  CTC  and  TLs  and  18%  for  TMs). 

Some  relevant  considerations  may  be  raised: 

•  Availability  of  all  websites  (for  EDGE  and  COLLABORATIVE)  may  have 
contributed  to  less  P2P  interactions.  Websites  are  in  fact  efficient  mechanisms 
to  share  information,  replacing  need  of  shares.  An  interesting  aspect  to  further 
analyze  is  the  effect  (positive  or  negative)  of  technologies  w.r.t.  mediating  (or 
replacing)  interpersonal  interactions. 

•  Moreover,  EDGE  registered  a  higher  value  for  network  exclusion  than 
COLLABORATIVE,  meaning  that  moving  from  an  ordered  structure  (teams 
and  team  roles)  to  a  flat  one  may  also  reduce  P2P  interactions:  in 
COLLABORATIVE,  subjects  were  given  instructions  to  support  other 
members,  but  in  EDGE  they  weren’t. 

•  Another  interesting  aspect  arises  when  analyzing  the  average  network  reach  for  each  approach:  all  approaches, 
except  CONFLICTED,  have  a  value  close  to  17%.  Clearly,  there  is  a  gap  between  potential  and  true  reach  of 
network,  which  tends  to  increase  with  network  size.  Moreover,  what  is  the  average  network  reach  for  humans? 

More  data  and  research  is  required  to  properly  address  these  considerations. 

The  final  metric  obtained  to  characterize  and  evaluate  interactions  in  ELICIT  is  the  Quality  of  Interactions  (QI), 
which  was  measured  using  following  arithmetic  function  as  follows: 


Run  ID 

Average 

Min 

Max 

Nbr<  10% 

Nbr  >  50  % 

LI-01 

15% 

0% 

18% 

2 

0 

LI -02 

13% 

0% 

18% 

3 

0 

LI -03 

15% 

0% 

18 

1 

0 

L2-01 

18% 

12% 

24% 

0 

0 

L2-02 

13% 

6% 

24% 

6 

0 

L2-03 

16% 

6% 

24% 

3 

0 

L2-04 

17% 

12% 

24% 

0 

0 

L3-01 

18% 

12% 

24% 

0 

0 

L3-02 

17% 

6% 

24% 

3 

0 

L3-03 

15% 

6% 

24% 

2 

0 

L3-04 

19% 

12% 

24% 

0 

0 

L4-01 

17% 

0% 

41% 

6 

0 

L4-02 

21% 

0% 

59% 

5 

1 

L4-03 

20% 

0% 

47% 

4 

0 

L4-04* 

13% 

0% 

35% 

8 

0 

L5-01 

26% 

6% 

65% 

4 

2 

L5-02 

14% 

0% 

35% 

8 

0 

L5-03 

9% 

0% 

71% 

14 

1 

Table  8  -  Network  Reach  Values 


Q Interactions  _  factoids 


shared  _  and  _  posted 


-  y  yV  _  factoids 


shared  _  and  _  posted 


QI  is  positive  if  more  relevant  factoids  are  shared  and  posted  than  non-relevant  ones  and  negative  otherwise.  Note 
that  quantity  matters  when  obtaining  QI.  Results  are  presented  in  Figure  1 1  (detailed  values  are  presented  in  Table  30 
in  Annex). 

COLLABORATIVE  was  the  most 
successful  approach  filtering  and 
distributing  relevant  information  over  non- 
relevant  information,  obtaining  highest  QI 
value  (45,  with  a  high  min-max  variation), 
followed  by  EDGE  (27).  All  other  three 
approaches  achieved  low  scores. 
DECONFLICTED  had  a  wide  min-max 
variation  (from  -15  to  52). 

QI  increases  as  maturity  approach 
increases  until  COLLABORATIVE  and 
decreases  in  EDGE. 


Quality  of  Interactions 


Figure  11  -  Quality  of  Interactions 
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Cognitive  Domain 

The  cognitive  domain  deals  with  subjects’  awareness  and  understanding  of  problem.  Its  complex  processes  occur 
within  subjects’  brain,  but  its  outcomes  may  be  measured  when  an  identification  attempt  is  performed. 

A  key  variable  is  extent  of  correct  understanding,  which  measures  the  number  of  correct  identifications  in  each 
problem  space  (i.e.,  who,  what,  where  and  when).  The  overall  results  per  approach  are  presented  in  Figure  12  and 
Figure  13  further  details  score  per  problem  space.  Detailed  data  is  presented  in  Table  34). 


Figure  12  -  Extent  of  Correct  Understanding 


In  overall,  extent  of  correct  understanding  was  low  (always  below 
30%).  EDGE  obtained  the  overall  best  score  (22.6  correct 
answers),  closely  followed  by  COLLABORATIVE  (20.8  correct 
answers). 

Somewhat  surprising  was  the  low  value  obtained  for  COORDINATED  (7.5  correct  answers).  It  seems  its  centralized 
nature  degraded  overall  extent  of  correct  understanding,  being  surpassed  by  DECONFLICTED  (11.2  correct  answers, 
although  with  a  large  min-max  range). 

CONFLITED  results  are  consistent  with  the  model:  it  obtained  the  worst  scores  (on  average,  only  4.7  correct 
answers). 

Extent  of  correct  understanding  was  further  decomposed  per  problem  space,  providing  further  evidence  toward 
assessing  an  approach  effectiveness  across  in  each  space.  Results  are  illustrated  in  Figure  13.  The  findings  are 
consistent  across  all  approaches:  in  terms  of  success  rate,  WHO  and  WHEN  had  the  highest  scores  (easiest  to  solve) 
and  WHAT  had  the  lowest  scores  (most  difficult  to  solve). 

Moreover,  scores  for  the  extent  of  correct  understanding  were  awarded  to  each  run  with  timestamp  (indicating  when 
score  was  achieved).  Results  are  provided  in  Table  9.  Only  COLLABORATIVE  and  EDGE  reach  30%  scores,  being 
EDGE  always  faster  (30%@1258  seconds  for  L4-02).  Most  COLLABORATIVE  runs  only  reached  20%. 
COORDINATED  never  reached  more  than  10%  scores,  DECONFLICTED  did  better  with  2  runs  at  20%  and 
CONFLICITED  never  reached  minimum  score. 

The  last  variable  used  to  assess  C2  approaches  in  the  cognitive  domain  is  a  measure  of  self-synchronization,  labeled 
as  Cognitive  Self-Synchronization. 


Figure  13  -  Extent  of  Correct  Understanding  per  Problem  Space 


Extent  of  Correct  Shared  Understanding 


Run  ID 

-1 

LU 

> 

LU 

-1 

10% 

20%  30%  50%  60%  80%  100% 

LI-01 

LI 

LI -02 

LI 

LI -03 

LI 

L2-01 

L2 

L2-02 

L2 

L2-03 

L2 

1028 

1618 

L2-04 

L2 

2120 

2212 

L3-01 

L3 

L3-02 

L3 

1509 

L3-03 

L3 

2058 

L3-04 

L3 

1840 

L4-01 

L4 

1874 

2107 

L4-02 

L4 

1353 

1760  2010 

L4-03 

L4 

1831 

2331 

L4-04 

L4 

1361 

1880 

L5-01 

L5 

1053 

1892 

L5-02 

L5 

977 

1101  1258 

L5-03 

L5 

1472 

1583  1705 

Table  9  -  Extent  of  Correct  Shared  Understanding  Scores 
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The  method  used  to  measure  this  variable  was  inspired  by  Moffat’s  work  towards  developing  a  knowledge  metric 
(Moffat  2003)  to  measure  the  amount  of  uncertainty  in  a  probability  distribution  (based  on  Shannon’s  Information 
Entropy). 

Cognitive  Self-Synchronization  (CSSync)  will  measure  the  amount  of  disorder  (or  entropy)  of  an  organization 
towards  determining  the  problem  (i.e.,  finding  who ,  what ,  where  and  when).  Note  that  whether  the  solution  is  correct 
or  wrong  is  not  relevant  here.  Translating  subjects’  identifications  as  a  measure  of  uncertainty,  the  function  we  will 
use  to  represent  CSSync  is  the  following: 

C  S  Sync  probiemSpace  ^  -  X  P(S'  )  *  ln(PU  ))  1  MClX  -  DiSOrderproblemSpace 

i= 1 

CSSync  is  measured  for  each  identification  input  field17  (or  ProblemSpace ),  namely,  who ,  what ,  where ,  when  (further 
decomposed  into  when-time,  when-day  and  when-month ),  in  which: 

•  N  refers  to  total  number  of  unique  identifications  (in  the  corresponding  ProblemSpace). 

•  P(Si)  is  the  proportion  of  a  given  identification  St  relative  to  organization  size  (size  is  equal  to  17,  i.e.,  the  total 
number  of  subjects  in  ELICIT).  Note  that  0  <  P  <  1 . 

•  Max_Disorder probiemspace  refers  to  maximum  entropy  value  (described  below)  and  is  used  to  normalize  value 
(between  0  and  -1). 

Addition  of  1  is  done  so  that  CSSync  e  [04].  The  value  is  interpreted  as  follows: 

•  CSSync=0  means  system  is  fully  disordered. 

•  CSSync=l  means  system  is  fully  synchronized. 


We  assume  that  any  organization  operating  in  ELICIT  has  an  initial  state  of  maximum  disorder  (maximum  entropy), 
that  is: 

P(S)  =  —  (since  all  N  subjects  have  different  understandings,  there  are  N  possible  outcomes  each  with  equal 
N 


probability  or  proportion,  where  N=  17)  and,  therefore,  Max  _  Disorder problemSpace 


N  N 


The  measure  for  the  overall  organizational  CSSync  is  simplified  to  be  the  sum  of  the  partial  CSSync  ProblemSpace  that  is: 
CSSync=0.25  *  ^  CSSynci  (weights  are  used  to  normalize  total  CSSync) 

i= problemSpace 

As  game  progresses,  subjects  make  identification  attempts  and  it  is  expected  that  some  of  these  attempts  are 
equivalent.  In  such  scenario,  the  cognitive  disorder  decreases  (or,  the  cognitive  order  increases)  and  the  organization 
is  said  to  be  converging  to  a  common  understanding  of  the  problem.  Ultimately,  if  all  subjects  provide  the  same 
identifications,  we  may  conclude  that  the  system  was  able  to  fully  self-synchronize  and  converge.  In  such  scenario: 

P ( ^probiemSpace ) =  1  f or  all  ProblemSpace,  (all  subjects  have  the  same  understanding  of  the  problem)  and, 
therefore,  CSSync  =  1 . 

Note  also  that,  if  subjects  don’t  perform  an  identification  attempt,  disorder  is  assumed. 

Figure  14  presents  the  overall  results  for  CSSync  across  the  C2  approaches  (for  detailed  data  see  Table  32  and  Table 
33  in  Annex). 

A  first  conclusion  is  that  cognitive  disorder  decreases  as  the  maturity  increases: 

•  CONFLICTED  is  the  most  disordered  state  (with  low  variation  from  average), 

•  DECONFLICTED  and  COORDINATED  have  similar  mean  values,  but  the  former  presents  a  larger  variation 
from  the  average  value  than  the  later. 

•  EDGE  reached  most  synchronized  state  (0.42  for  L5-01)  and  overall  best.  This  finding  is  strengthened  by  the 
small  min-max  range  variation. 

•  COLLABORATIVE  second  best  synchronized  state  (0.34)  but  with  a  min-max  variation  (between  0.22  and  0.42). 


17  In  ELICIT,  Subjects  enter  identification  attempts  in  free  forms  for  the  following  fields:  who,  what,  where,  when-time,  when-day,  when-month. 
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It  is  interesting  to  note  that,  although 
both  EDGE  and  COLLABORATIVE 
provided  same  access  to  information 
and  subjects,  the  degree  of  cognitive 
self-synchronization  achieved  was 
lower  for  the  later.  The  fact  that  in 
COLLABORATIVE  there  was  an 
assignment  of  roles  (i.e.,  CTC,  team 
leaders  and  team  members)  may  have 
induced  less  pro-activity  in  team 
members  to  work  toward  finding  the 
solution  while,  on  the  other  hand, 
adoption  of  power-to-the-edge 
principles  (Alberts  and  Hayes  2003) 
in  EDGE  may  have  increased 
members’  pro-activity  to  work 
towards  finding  the  solution  resulting 
in  an  increase  in  overall  (cognitive) 
self-synchronization. 

Charts  displaying  the  evolution  of  CSSync  through  time  are  presented  in  Figure  28  in  Annex.  A  remarkable  finding  is 
observing  the  early  disorder  decrease  in  EDGE  run  L5-02  (0.25@1250sec)  while,  for  all  other  approaches,  this  occurs 
at  late  stages  of  the  experiment. 

Measures  of  Merit 

Assessment  of  approaches  is  made  with  basis  on  the  following  Measures  of  Merit  (Alberts  and  Hayes  2002): 

•  Measures  of  Force  Effectiveness  (MoFE):  will  measure  organization  effectiveness  with  basis  on  the  criteria 
defined  in  Table  5. 

•  Measures  of  Performance  (MoP):  will  measure  efficiency  given  effectiveness  in  two  dimensions: 

o  Measure  of  time  (e.g.,  how  fast)  to  reach  a  given  effectiveness  level, 
o  Measure  of  effort  (e.g.,  transactions  costs)  to  reach  a  given  effectiveness  level. 

Organization  Effectiveness  (MoFE)  is  determined  according  to  effectiveness  criteria  described  in  Table  5.  Since  there 
are  4  problem  spaces,  the  following  formula  is  used: 

Effectiveness  =^0.25  *  Correct _anst ,  in  which: 

i 

•  i  is  the  index  of  each  problem  space. 

•  If  i =who,  what  and  where :  Correct_anSi  is  1.0  if  correct  answer  is  provided  and  0.0  otherwise. 

•  If  i =when:  Correct _ans when  =034* Correct _answheit  time  +  033*  Correct  _ans  whenday  +0.33* Correct- ans when_month 


Figure  14  -Cognitive  Self-Synchronization  (Cognitive) 


Overall  scores  are  presented  in  Figure  15  (detailed  data  is  provided  in  Table  35  in  Annex). 

COLLABORATIVE  approach  had  best  (100%)  and  overall  best  score  (80%),  EDGE  was  the  second  best  (61%). 
COORDINATED  (with  34%)  was  again  surpassed  by  DECONFLICTED  (44%)  and  CONFLICTED  had  worst  score 
(31%). 

In  fact,  the  COORDINATED  approach  is  an  interesting  case  to  analyze  further.  Although  Coordinator  (CTC)  had 
most  information  available,  he  didn’t,  for  most  runs,  provide  a  correct  solution.  Furthermore,  in  run  L3-01,  CTC 
didn’t  performed  any  ID  (0  score  was  given  as  a  consequence),  in  L3-02  and  L3-03  scores  were  low  (0.25  and  0.17 
respectively),  but  L3-04  achieved  a  high  score  of  92%.  Effectiveness  in  a  COORDINATED  approach  is  highly 
dependent  on  the  performance  of  the  coordinator. 
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Another  surprising  result  was 
the  effectiveness  decrease  from 
COLLABORATIVE  to  EDGE. 
Three  main  factors  are 
speculated  to  contribute  to  this 
result: 

1.  Non-task  specialization  (i.e., 
all  subjects  work  in  all  spaces) 
makes  problem  more  difficult. 
On  the  other  hand, 
COLLABORATIVE  set  teams 
assigned  to  tasks  (i.e.,  solve  a 
problem  space)  with  positive 
results. 

2.  Information  overload:  after 
an  EDGE  run,  several  subjects 
(but  not  all)  complained  about 

having  too  much  information  to  process. 

3.  Organizational  effectiveness  criteria:  the 
criteria  used  for  EDGE  was  clearly  the  most 
demanding  (the  answer  considered  was  the 
mode  value  taken  from  the  subjects  answers 
set  in  each  problem  space)  and  it  never  was 
achieved  when  composite  answers  were 
required  (typical  in  WHAT).  For  example, 
L5-02  and  L5-03  used  factoid  set  4,  in  which 
correct  answer  for  what  is  ‘SECULAR 
SCHOOL’,  but  most  subjects  answered 
‘SCHOOL’  resulting  in  failure.  As  illustrated 
in  Figure  16,  no  EDGE  run  succeeded  finding 
a  correct  what.  Also,  conducting  two  runs 
with  factoid  set  4  may  have  introduced  some 
bias  in  the  effectiveness  results. 
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Figure  16  -  Organization  Effectiveness  per  solution  space 


Organization  efficiency  (MoP)  is  determined  to  assess  organization  performance  in  terms  of  time  efficiency  (given 
effective)  and  effort  spend.  Two  variables  are  used:  time-efficiency  and  effort-efficiency. 


To  calculate  time-efficiency,  the  following  formula  was  used: 

Efficiency time  =  Effectiveness  _  score 2  *  log10  (1  +  ^  m) 

The  formula  awards  higher  values  for  organizations  which  are  more  effective  and  faster  reaching  the  last  correct 
identification  and  penalize  those  which  are  less  effective  and  slower.  Figure  17  presents  the  results  per  run  (left)  and 
mean  values  per  approach  (right)  in  a  normalized  scale  (0-1).  Data  for  calculations  was  extracted  from  Table  35. 
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Figure  17  -  Time  Efficiency:  per  run  (left)  and  per  approach  (right) 
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In  general,  increasing  the  maturity  approach  results  in  a  time-efficiency  increase  (with  a  slight  decrease  from 
COLLABORATIVE  to  EDGE).  Yet,  results  obtained  were  irregular  across  runs: 

•  Most  time-efficient  run  was  L4-01  (COLLABORATIVE)  and  second  best  was  L3-04  (COORDINATED). 
Two  EDGE  runs  followed  (L5-02  and  L5-03). 

•  COORDINATED  had  three  runs  with  extremely  low  time-efficiency  (L3-01  being  the  lowest).  It  should  be 
also  mentioned  the  low  score  of  EDGE  run  L5-01  (with  low  effectiveness  score). 

To  calculate  effort-efficiency,  the  following  formula  was  used: 

Efficiency effort  =  Effectiveness  _  score 2  *  log10  (1  +  fyort  spent) 

effort  spent  accounts  all  transactions  costs  occurring  in  a  ELICIT  run,  in  which  a  share ,  post ,  pull  or  identification 
action  is  considered  to  have  a  (unitary)  transaction  cost. 

The  formula  awards  higher  values  for  organizations  which  are  more  effective  and  spend  less  (i.e.,  less  transactions) 
during  the  run  and  penalize  those  which  are  less  effective  and  spend  more  (i.e.,  more  transactions). 

Figure  17  presents  the  results  per  run  (left)  and  mean  values  per  approach  (right)  in  a  normalized  scale  (0-1).  Data  for 
calculations  was  extracted  from  Table  35  (effectiveness  score)  and  Table  26  (effort).  Additionally,  Figure  29  in  annex 
presents  the  average  value  of  effort  spent  per  approach:  clearly,  EDGE  spent  more  effort  than  any  other  approach 
(more  than  400  units  than  COLLABORATIVE,  the  second  more  expensive),  while  CONFLICTED  spent  less.  Also 
interesting  was  having  COORDINATED  as  second  less  expensive  approach. 
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Figure  18  -  Effort  Efficiency:  per  run  (left)  and  per  approach  (right) 


In  general,  increasing  the  maturity  approach  results  in  a  effort-efficiency  increase,  except  when  moving  from 
COLLABORATIVE  to  EDGE,  in  which  a  significant  decrease  was  observed  (less  efficient  than  COORDINATED 
and  as  inefficient  as  DECONFLICTED).  Yet,  results  obtained  were  irregular  across  runs: 

•  Most  effort-efficient  run  was  L3-04  (COORDINATED)  and  second  best  was  L4-01  (COLLABORATIVE) 
followed  by  L4-03. 

•  Ironically,  it  was  L3-01,  a  COORDINATED  run,  which  achieved  lowest  effort-efficiency.  In  fact,  all  other 
COORDINATED  runs  had  low  scores. 

The  low  effort-efficiency  obtained  for  EDGE  is  justified  by  the  large  effort  spent  by  this  approach,  mainly  due  to  the 
massive  use  of  pulls.  However,  there  should  be  caution  interpreting  effort-efficiency  considering  current  experiments’ 
setup:  no  indication  of  cost  to  deduct  any  organizational  goals  was  provided  in  subject’s  instructions  and,  therefore,  it 
was  not  a  concern  for  subjects  to  save  actions. 
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CONCLUSIONS 

The  validation  of  the  N2C2M2  resorting  to  the  ELICIT  platform  and  a  set  of  experiments  specifically  designed  to 
model  each  approach  level  yielded  interesting  results  and  enabled  a  deeper  understanding  of  maturity  model 
approaches  and  related  assumptions  (or  hypothesis).  This  work  also  focused  on  measuring  key  variables  so  that  a 
quantitative  analysis  could  be  conducted. 

The  overall  assessment  of  C2  approaches,  according  to  the  results  obtained  in  the  ELICIT  experiments,  is  presented  in 
Table  10.  C2  approaches  are  evaluated  in  a  1-5  scale:  1  refers  to  best  score  and  5  refers  to  worst  score.  Grey 
background  indicates  non-compliance  with  model  hypothesis. 


C2  Approach 

Domain  /  Variable  Assessed 
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2 
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Table  10  -  Overall  Position  of  C2  Approaches  across  assessed  variables 


In  overall,  the  results  are  consistent  with  model  expectations,  but  some  deviations  are  observed  especially  regarding 
some  assessment  scores  between  EDGE  and  COLLABORATIVE  and  COORDINATED  and  DECONFLICTED. 

We  start  by  analyzing  these  results  together  with  verification  of  hypothesis  [4],  [5]  and  [6],  which  focused  on  the 
model  intermediate  variables  (Figure  2). 

[4]  Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of  Quality  of  Individual  and  Shared 
Information  than  lower  collective  C2  maturity  approaches. 

Results  obtained  for  all  approaches  were  consistent  with  this  hypothesis.  More  specifically,  regarding  shared 
information  reach  across  organization,  only  EDGE  and  COLLABORATIVE  achieved  non-zero  scores  (with  high 
levels).  Relevant  information  reached  for  key-roles  (i.e.,  CTC  and  TLs)  was  therefore  measured  and  it  was  observed 
that  COORDINATED  obtained  high  levels  for  CTC  but  low  levels  for  TLs,  meaning  coordinator  didn’t  effectively 
shared  information  with  its  ‘subordinates’.  Relevant  information  reached  decreased  for  DECONFLICTED  (steep 
decrease  for  CTC)  and  reached  a  minimum  at  CONFLICTED. 

An  aspect  worth  to  note  is  that  no  organization  reached  100%  of  information  shared.  In  fact,  local  hoarding  of  critical 
information  across  runs  and  approaches  was  common  (such  behavior  is  inconsistent,  for  example,  with 
COLLABORATIVE  and  EDGE  approaches). 

While  EDGE  approaches  were  the  more  effective  (and  efficient)  distributing  information,  best  efficiency  score  was 
achieve  by  COLLABORATIVE  (L4-02)  and,  regarding  individual  reach  (CTC),  COORDINATED  (L3-02). 

[5]  Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of  Quality  of  Individual  and  Shared 
Awareness  and  Understanding  than  lower  collective  C2  maturity  approaches. 

Results  obtained  for  all  approaches,  except  COORDINATED,  were  consistent  with  this  hypothesis.  We  confirmed 
that  levels  of  Individual  and  Shared  Awareness  and  Understanding  increased  from  CONFLICTED  (lowest  value),  to 
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DECONFLICTED,  COLLABORATIVE  and  EDGE.  However,  COORDINATED  approach  had  lower  values  than  for 
DECONFLICTED.  As  noted  before,  the  COORDINATED  approach  empowers  a  centralized  node  (CTC),  hence 
concentrates  power  and  knowledge,  with  negative  influence  on  situation  understanding  for  all  other  nodes.  The  fact 
that  DECONFLICTED  sets  independent  decision  making  in  each  team,  hence  distributing  cognitive  goals  and  efforts, 
may  explain  the  third  best  score. 

[6]  Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of  Self-Synchronization  (at  cognitive 
level)  than  lower  collective  C2  maturity  approaches. 

Results  obtained  for  all  approaches  were  consistent  with  this  hypothesis.  An  aspect  to  note  was  the  high  value 
obtained  for  EDGE  and  small  min-max  variation.  Moreover,  EDGE  was  also  the  fastest  approach  to  self-synchronize. 

[1]  For  a  complex  endeavor,  higher  collective  C2  maturity  approaches  are  more  effective. 

Results  obtained  were  consistent  with  this  hypothesis  for  all  approaches  except  COORDINATED  and  EDGE. 
COLLABORATIVE  was  the  most  effective  approach,  followed  by  EDGE.  EDGE  approach  was  particular  sensitive 
to  WHERE  problem  (no  correct  solution  was  ever  found  at  organization  level).  Moreover,  DECONFLICTED 
achieved  better  scores  than  COORDINATED.  The  later  was  particularly  irregular,  for  overall  results  were  mediocre 
(below  0.25),  except  for  L3-04  that  was  almost  100%  effective.  In  fact,  CTCs  had  always  access  to  information 
necessary  to  solve  one  or  more  parts  of  the  problem,  meaning  conditions  to  succeed  were  present,  but  not  having  the 
right  subject  assigned  to  the  CTC  role  resulted  in  failure.  We  consider  this  analogy  valid  to  real  organizations  that 
adopt  COORDINATED  approaches,  i.e.,  organizational  success  is  heavily  dependent  on  a  single  person’s  judgments 
and  decisions,  and  ability  to  remain  adequate  to  a  wide  range  of  circumstances  and  complex  endeavors. 

[2]  For  a  given  level  of  effectiveness,  higher  collective  C2  maturity  approaches  are  more  efficient. 

Efficiency  was  measured  as  a  function  of  effectiveness  in  two  dimensions:  time  and  effort. 

Time-efficiency  results  obtained  were  consistent  with  this  hypothesis  for  all  approaches  except  for  EDGE,  since  its 
value  was  below  COLLABORATIVE.  Although  EDGE  reached  a  high  time-efficiency  in  two  runs,  overall  result  was 
penalized  due  to  a  bad  run  (L5-01). 

Effort-efficiency  results  obtained  were  consistent  with  this  hypothesis  for  all  approaches  except  for  EDGE,  since  its 
value  was  below  COLLABORATIVE  and  even  COORDINATED.  In  fact,  a  surprising  result  was  having  a 
COORDINATED  run  (L3-04)  obtaining  highest  effort-efficiency  (despite  the  fact  that  all  other  runs  obtained  very  low 
efficiency  values  as  a  result  of  their  effectiveness).  Moreover,  effort  was  measured  per  approach  and 
COORDINATED  spent  less  than  DECONFLICTED,  COLLABORATIVE  and  EDGE.  Hence,  if  effective,  a 
COORDINATED  approach  may  be  very  efficient  regarding  effort  spent.  On  the  other  hand,  EDGE  always  spent 
more  effort  than  other  approaches.  Hence,  a  single  low  effective  run  degraded  overall  EDGE  cost-efficiency  results. 

[3]  Higher  collective  C2  maturity  approaches  are  more  agile. 

Given  its  novelty,  Agility  concept,  and  its  application  in  organizations,  is  currently  under  analysis  by  researchers  and 
groups,  such  as  CCRP  FACT18  group  and  NATO  SAS-08519.  Therefore,  we  deemed  essential  to  consider  outputs 
from  these  groups  before  attempting  to  capture  and  measure  Agility  in  ELICIT. 

This  hypothesis  therefore  will  be  covered  by  future  research  work. 

[7]  Organizations  require  a  minimum  level  of  maturity  to  be  effective  in  ELICIT. 

Effectiveness  of  the  several  approaches  in  ELICIT  varied  across  runs: 

•  Only  COLLABORATIVE  and  EDGE  reached  an  average  value  above  50%  effectiveness. 
COLLABORATIVE  presented  the  best  results  in  a  consistent  way  (80%  effectiveness,  ranging  between  67% 
and  100%). 

•  A  COORDINATED  run  reached  a  high  mark  of  92%,  despite  the  fact  that  all  other  runs  had  a  mediocre  score 
and  average  score  was  34%. 

•  Both  CONFLICTED  and  DECONFLICTED  reached  low  average  scores. 

This  suggest  that  requisite  maturity  in  ELICIT  is  COLLABORATIVE  or  COORDINATED  whenever  a  fit  subject  is 
placed  in  CTC  role. 


18  Information  about  FACT  (Focus,  Agility,  and  Convergence  Team)  is  available  at  http://www.dodccrp.org/html4/events_fact.html 

19  NATO  SAS-085  (C2  Agility  and  Requisite  Maturity)  official  page:  http://www.rta.nato. int/ACTIVITY  META.asp?ACT=SAS-085 
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[8]  Increasing  the  degree  of  difficulty  in  ELICIT  require  organizations  to  increase  their  level  of  maturity  to 
maintain  effectiveness  in  ELICIT. 

Evaluation  of  this  hypothesis  required  running  factoids  set  2  (most  difficult  set)  across  all  approaches  and  determine 
respective  impact  on  effectiveness.  This  condition  was  not  met  and,  therefore,  this  hypothesis  cannot  be  evaluated. 

Finally,  an  interesting  fact  to  derive  from  Table  10  is  that  C2  approaches  assessment  scores  are,  in  overall,  consistent 
with  N2C2M2  model,  NCW  tenets  and  NCW  value  chain:  increasing  the  C2  approach  maturity  of  an  organization, 
increases  the  extent  of  shared  information  and  critical  information  available,  result  in  better  interactions,  broader 
extent  of  correct  understanding,  self-synchronization  and  increased  organizational  effectiveness  and  efficiency. 

We  consider  that  there  are  still  aspects  to  further  investigate  and  understand  regarding  validation  of  the  N2C2M2,  such 
as  achieving  a  better  understanding  of  the  non-compliances  between  the  N2C2M2  model  and  some  experimentation 
results  (especially  regarding  EDGE  and  COORDINATED  approach). 

Moreover,  there  is  a  clear  need  to  increase  the  amount  of  experimental  data  (i.e.,  valid  ELICIT  runs)  for  analysis  to 
ensure  robustness  of  results  and  findings  obtained  so  far.  In  this  direction,  the  agent-based  ELICIT  (Ruddy  2009) 
presents  a  cost-effective  way  to  replicate  experiments  described  herein,  but  resorting  to  software-agents  instead  of 
humans,  and  exploit  further  manipulations  in  the  context  of  the  N2C2M2  and  NCW.  Software-agents  are  fully 
controllable  and  it  is  easier  to  collect  17  of  them  to  run  an  experiment,  although  at  the  cost  of  losing  the  richness  of  the 
human  complexity  in  the  game.  Hence,  we  intend  to  complement  agent-based  simulated  runs  with  runs  using  real- 
persons  in  our  ELICIT  experiments. 
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Annexes 

OVERVIEW  OF  THE  ELICIT  PLATFORM 

ELICIT  is  a  research  and  experimentation  programme  developed  for  the  CCRP  to  conduct  research  related  with 
collaboration,  information  sharing  and  trust.  The  platform  allows  instantiating  different  C2  approaches  and 
observation  of  behaviors  and  dynamics  in  the  information,  cognitive  and  social  domains.  ELICIT  is  presented  in 
detailed  in  (Ruddy  2008). 

Logic  of  Factoids  Sets 

A  Factoid  is  a  piece  of  information  which  may  contain  important  information  to  determine  the  Who,  What ,  Where  and 
When  of  a  future  attack.  Factoids  are  categorized  as: 

Key  (K):  if  they  contain  information  which  is  essential  for  a  specific  problem  space. 

Expertise  (E):  if  they  contain  information  which  is  essential  for  solving  the  problem  and  may  be  important  for 
more  than  one  specific  problem  space.  Expertise  factoids  model  the  expertise  that  a  team  leader  might  have. 
Supportive  (S):  if  they  contain  information  which  supports  key  and  expertise  facts. 

Noise  (N):  if  the  information  they  contain  is  irrelevant  to  determine  the  problem. 

ELICIT  has  four  baseline  sets  of  factoids,  each  consisting  on  a  different  problem  to  solve.  They  should  comply  with 
the  following  generic  rule:  each  factoid  set  has  68  factoids,  whose  category  is  distributed  as  follows:  17  are  Key  and 
Expertise,  17  are  Supportive  and  34  are  Noise.  Hence,  ratio  of  relevant  information  and  noise  is  50%. 

All  factoids  are  distributed  by  server  in  three  waves  as  follows: 

Wave  1  (t=0  min):  34  factoids  are  distributed  by  server  (each  subject  receives  2  factoids),  in  which  9  are  K/E, 
9  are  S  and  16  are  N.  Special  roles  (e.g.,  team  leaders  and  coordinator),  if  applicable,  should  receive  an 
expertise  factoid  but  an  unintended  deviation  was  produced  due  to  new  organization  setup  (see  remark). 

Wave  2  (t=5  min):  17  factoids  are  distributed  by  server  (each  subject  receives  1  factoid),  in  which  8  are  K/E/S. 
Wave  3  (t=10  min):  17  factoids  are  distributed  by  server  (each  subject  receives  1  factoid),  in  which  8  are 
K/E/S. 

However,  in  the  scope  of  this  work,  all  factoids  were  analyzed  and  reclassified  resulting  in  factoids  sets  displaying 
minor  deviations  from  this  generic  rule.  This  is  explained  in  detailed  in  following  subsections. 

Next  chart  illustrates  factoids  distribution  across  time  for  an  Edge  run: 


Figure  19  -  Information  Accessible  Chart  (server  factoids  distribution) 


Likewise,  the  following  table  presents  factoids  distribution  for  an  Edge  run  with  added  feature  to  discriminate  which 
factoid  is  distributed  to  which  subject. 
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Figure  20  -  Information  Accessible  Table  (server  factoids  distribution) 


Analyzing  both  figures  it  is  possible  to  confirming  that: 

1.  All  information  (i.e.,  68  factoids)  is  accessible  at  6000  sec  (10  min) 

2.  Server  only  distributes  unique  factoids  to  subjects,  that  is,  it  doesn’t  disseminate  the  same  factoid  to  more  than 
one  subject  (in  other  words,  shared  information  resulting  from  server  factoids  distribution  is  always  zero 
between  any  set  of  subjects). 

3.  K/E,  S  and  N  factoids  are  evenly  distributed  to  subjects  (except  cases  noted  in  the  beginning  of  this  section) 
Examples  of  factoids: 

Key  factoid  (who):  The  Lion  is  known  to  work  only  with  the  Azure,  Brown,  or  Violet  groups 
Expertise  factoid  (what):  There  will  be  a  suicide  bomber  attack  at  a  school 

Supportive  factoid  (when):  The  Blue,  Silver,  Turquoise  and  Gray  groups  prefer  to  attack  in  daylight 


Solution  Logic 

All  factoids  sets  have  different  factoids  and  solutions.  The  solution  logic  is  similar  for  all  sets  and  also  follows  a 
similar  (but  not  equivalent)  distribution  in  factoids’  classification. 


Factoid  Set  Nbr 

Nbr  Factoids 

Nbr  K/E 

Nbr  S 

Nbr  N 

KE  ratio 

KES  ratio 

Noise  ratio 

1 

68 

16 

18 

34 

0.24 

0.50 

0.50 

2 

68 

17 

18 

33 

0.25 

0.51 

0.49 

3 

68 

15 

20 

33 

0.22 

0.51 

0.49 

4 

68 

15 

20 

33 

0.22 

0.51 

0.49 

Table  11  -  Factoids  classification  distribution  per  set 


The  solution  logic  is  explained  in  next  subsections. 
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1 . 1 . 1  Factoid  Set  1 

Solution  is: 

-  Who:  Violet 

What:  Financial  Institution 
Where:  Psiland 

-  When:  April,  5,  11:00AM 

The  next  table  presents  factoids  that  allow  to  unequivocally  determining  each  part  of  the  solution: 


\  Factoid 

\  Domain 

Possible 
Solutions  Set 

Who 

What 

Where 

When 

OBS: 

Solution  \ 

Domain  \ 

Who 

Azur,  Brown, 
Coral,  Chartreuse, 
Gold,  Purple, 
Violet 

1,8,10,13,14 

1,8:  is  A,  B,  V 

10:  not  A 

13,14:  not  B 

What 

Embassies  (20x) 
Dignitaries 

Fin.  Inst. 

2,  22,  29 

39,41,42,47 

42:  is  E,  D,  IF 

29:  not  D 

2,  22,  39,  41:  all  embassies  out 

Where 

Tauland 
Epsilonland 
Chiland  Psiland 
Omegaland 

1 

3,  22,  42,  47 

42:  provides  who  options 
(T,E,C,P,0) 

22:  not  T,  not  E 

47:  not  O 

1,3:  not  C 

When 

Month:  April,  June, 
any 

Day:  5,10 

T:  11:00  AM 

1,8 

53,  59,  62 

1,8:  Lion  is  involved  and  is  a 
person 

53:  is  11:00AM 

59:  is  April 

1,8,62:  is  day  5 

Modified  factoid  53  specifies 

AM  (day). 

Table  12  -  Factoids  necessary  to  determine  solution  for  Set  1 


Remarks: 

•  41  is  ‘N’  but  should  be  K’ 

•  K/E  factoids  not  necessary:  4  (E),  31  (K) 

Factoids  Reclassification 


Original 

Modification 

4E 

4S 

31K 

31S 

41N 

41K 
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1 . 1 . 2  Factoid  Set  2 

Solution  is: 

Who:  Aqua 

What:  Deltaland  Embassy 
Where:  Alphaland 
-  When:  May,  27,  3:00PM 

The  next  table  presents  factoids  that  allow  to  unequivocally  determining  each  part  of  the  solution: 


\  Factoid 

\  Domain 

Possible 

Solutions  Set 

Who 

What 

Where 

When 

OBS: 

Solution  \ 
Domain  \ 

Who 

Aqua,  Orange, 
Indigo,  Lime, 
Marron,  Red,  Tan 

1,  12,  14,  17,  18 

1,  12:  is  A,  0,  T 

14:  not  T 

17,18:  not  O 

What 

Embassies  (20x) 
Dignitaries 

Fin.  Inst. 

2,  26,33,35 

43,46 

35:  provides  what  options  (D, 
Embassies) 

26:  not  A,B  targets 

33,  35:  not  Dignatary 

43 :  not  G-embassy,  not  B- 
embassy 

(can  only  be  D-embassy) 

Where 

Tauland, 
Epsilonland, 
Chiland,  Psiland  ou 
Omegaland 

1 

3,46,51 

46:  is  A,B,G,D,E 

1,3:  not  G,  not  D,  not  E 

5 1 :  not  B 

When 

Month:  May,  June 
Day:  16,  27 

T:  3:00PM,  night, 
day 

1,  12,  14 

55,  57,  63,  66 

1,12,14,55:  needs  who 

57:  is  3:00PM 

63 :  is  Maio 

66:  is  day  27 

Table  13  -  Factoids  necessary  to  determine  solution  for  Set  2 


Remarks: 

•  55  is  ‘N’  and  should  be  ‘K’  (55  replaces  17K  and  18K  -  actually,  the  fact  that  Orange  recruits  locals, 
doesn’t  mean  they  don’t  recruit  non-locals  also  -  as  such,  Lion  could  also  work  with  Orange) 

•  K/E  factoids  not  necessary:  4  (E) 

Factoids  Reclassification 


Original 

Modification 

4E 

4S 

55N 

55K 
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7. 7. 3  Factoid  Set  3 

Solution  is: 

Who:  Magenta 

What:  Southern  Oil  Pipeline  Terminal 

Where:  Thetaland 

When:  August,  15,  11:00PM 

The  next  table  presents  factoids  that  allow  to  unequivocally  determining  each  part  of  the  solution: 


\  Factoid 

\  Domain 

Solution  \ 
Domain  \ 

Possible 
Solutions  Set 

Who 

What 

Where 

When 

OBS: 

Who 

Beige,  Cyan,  Green, 
Fuchsia,  Ivory, 
Magenta,  Yellow 

1 

2,29 

59 

1,2,29:  is  M,C 

59:  not  C 

What 

Oil  Pipeline, 
Railway  station, 
Power  supply  station 

1 

2,21,23,  30 

23:  is  Oil, Train,  Power. 

1,2,  21:  not  Power 

1,2,30:  not  Train 

(54  and  67  indicates  a  ‘Southern 
pipeline’  but  it  isn’t  conclusive) 

Where 

Zetaland,  Thetaland, 
Iotaland,  Kappaland, 
Lambaland 

(needs  what) 

3,  40,  50 

23:  is  Z,T,I,K,L 

1 ,3 :  not  Z 

40  (with  what):  not  I 

50  (with  what):  not  K,L 

When 

Month:  June,  August 
Day:  14,  15 

T:  11.00PM,  night, 
day 

4,  54,61,67 

4,61:  is  1:00PM 

54,67:  is  15,  August 

Table  14  -  Factoids  necessary  to  determine  solution  for  Set  3 


Remarks: 

•  15  is  ‘N  and  should  be  ‘S’  (allows  to  eliminate  directly  an  option  -  and  discards  6K) 

•  6  is  ‘K’  and  should  be  ‘S’  (see  previous  point) 

•  59  is  ‘N’  and  should  be  ‘S’  (allows  to  eliminate  directly  an  option). 

•  K/E  factoids  not  necessary:  47  (K) 

•  Factoid  26N  is  never  disseminated20. 

This  factoid  set  doesn’t  provide  a  conclusive  solution  for  What.  Specifically,  it  is  clear  that  target  is  an  Oil  Pipeline, 
but  nowhere  is  explicated  that  it  is  the  Southern  pipeline  (see  factoids  54  and  67).  We  consider  that  solution  may  be 
obtained  through  logic  but  not  resorting  to  provided  factoids. 

Hence,  for  analysis  purposes,  we  consider  ‘Oil  pipeline’  as  a  correct  answer. 

Factoids  Reclassification 


Original 

Modification 

6K 

6S 

15N 

15S 

47K 

47  S 

59N 

59S 

20  factoidset3PT-17.txt  file  contains  variation  for  metadata  of  factoid  26  which  is  different  from  others.  Specifically: 
Correct  syntax:  “44|N|3|16|2|5|Zetaland  e  um  enclave” 

Incorect  syntax:  “26|N|2|16|34|4|Todas  as  centrais  electricas  em  Kappaland  sao  propriedade  do  Estado” 

The  fifth  element  (34)  in  factoid  26  refers  to  non-existing  dissemination  wave  (34). 
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1. 1. 4  Factoid  Set  4 

Solution  is: 

Who:  Magenta 
What:  Secular  School 
Where:  Omicronland 
When:  January,  1,  9:00AM 


The  next  table  presents  factoids  that  allow  to  unequivocally  determining  each  part  of  the  solution: 


\  Factoid 

\  Domain 

Solution  \ 
Domain  \ 

Possible 
Solutions  Set 

Who 

What 

Where 

When 

OBS: 

Who 

Blue,  Gray,  Silver, 
Teal,  Turquoise 

1,6,  10,  19 

2 

1,2:  not  G,  not  Teal 

2,6:  not  S,  not  R 

2,10:  not  Turq 

19  (OR  2,20):  not  B 

What 

Secular  school, 
Religious  school, 
Army  base 

2,  25,  27,  32, 

43,51 

2,32:  is  RSchool,  SSchool 

51:  not  School  in  S 

25,27,43:  not  RSchool 
(5 1  replaces  34) 

Where 

Muland,  Xiland, 
Omicronland,  Piland 
Sigmaland 

(who) 

2,  25,  27,  32 

3,38,43,51 

32:  provides  what  options 

43:  is  M,X,0,P,S 

2,  51:  not  S 

2,25,  38:  not  M 

2,27,  38:  not  X 

3  (with  who):  not  P 

When 

Month:  January 
Day:  1 

T:  9:00AM 

55,58,65 

55:  is  1 

58:  is  9:00AM 

65:  is  January 

Table  15  -  Factoids  necessary  to  determine  solution  for  Set  4 


Remarks: 

•  19  is  4 S’  but  could  be  ‘K’  (allows  to  eliminate  directly  an  option  -  although  key  factoid  20  allows  same 
inference  but  is  not  straightforward). 

•  All  K/E  factoids  are  necessary  (but  20K  may  be  replaced  by  19S) 

•  63  is  ‘N’  but  could  be  ‘S’  (moreover,  if  ‘needs  time’  is  interpreted  strictly,  could  eliminate  an  option  - 
Cyan  -  becoming  ‘K’) 

•  Factoid  65  is  incorrectly  numbered  as  56. 


Factoids  Reclassification 


Original 

Modification 

4E 

4S 

19S 

19K 

20K 

20S 

34K 

34S 

63N 

63  S 

65  S 

56S 
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DATA  COLLECTION  AND  ANALYSIS  PROCESS 

The  process  to  collect  and  analyze  experimental  data  in  ELICIT  underwent  the  following  stages: 

•  Setup  and  Preparation 

•  Experiment  Run 

•  Data  Collection 

•  Data  Fixes 

•  Data  Analysis 


The  process  is  depicted  in  Figure  21. 


Data  Analysis 


Figure  21  -  Data  Collection  and  Analysis  Process 


Setup  and  Preparation 

This  stage  consists  in  the  preparation  of  a  ELICIT  run,  which  include: 

•  Setting-up  and  configuring  the  platform  (N2C2M2  Moderator  Setup  Instructions) 

•  Providing  instructions  to  individuals 

o  Instructions  for  players  (per  C2  Approach) 
o  Presentation  material  (per  C2  Approach) 

•  Conducting  a  trial  run  for  training. 

Experiment  Run  and  Data  Collection 

This  stage  consists  in  the  actual  run  of  the  experiment  and  collection  of  data: 

•  The  ELICIT  platform  logs  actions  from  individuals: 

o  ELICIT  log  file  (plain  text  file). 

•  Individuals  may  make  notes  in  a  Scratch  paper,  which  is  collected  afterwards: 

•  Individuals  fill  a  questionnaire  at  the  end  of  the  experiment: 

Data  fixes 


This  stage  consists  in  fixing  data  collected  in  previous  phase.  For  example,  individuals  identifies  are  entered  as  free 
text  and  may  include  syntax  or  semantic  errors.  Therefore,  analysts  are  required  to  harmonize  inserted  text. 

Data  Analysis 


This  stage  consists  in  the  analysis  of  collected  data.  Automated  tools  will  be  use  to  generate  quantitative  data,  to  the 
extent  as  possible,  although  complementary  qualitative  analysis  is  also  required. 
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DETAILED  DESIGN  OF  EXPERIMENTS 

Hereby  is  presented  the  ELICIT  experimental  design  approach  used  in  a  set  of  ELICIT  runs  to  observe  the  N2C2M2 

Maturity  Levels.  It  starts  by  describing  what  is  common  to  all  levels  and  then  describes  the  differences  associated  with 

specific  maturity  levels. 

Common  Aspects 

The  following  aspects  are  common  to  all  levels: 

a)  Entities:  There  will  be  four  team  entities  and  a  fifth  single  entity  that  takes  on  different  roles  in  the  runs 
representing  Conflicted,  De-conflicted,  Coordinated  and  Collaborative  approaches  to  collective  C2.  In  the 
case  of  Edge  C2,  there  will  be  seventeen  entities  with  no  a  priori  collective  C2. 

b)  Context:  The  context  is  a  complex  endeavor  —  66 one  in  which  there  are  two  or  more  force  elements  (entities) 
present  and  where  one  of  more  of  the  following  conditions  exists:  the  entities  have  overlapping  intents;  the 
entities  are  operating  in  the  same  area  at  the  same  time;  and,  the  actions  taken  by  an  entity  can  come  into 
conflict  with  those  taken  by  another  entities .”  (Alberts  and  Hayes  2007). 

c)  Scenario:  The  scenario  will  be  the  existing  ELICIT  scenario  of  an  imminent  terrorist  attack.  Organic 
information  will  be  provided  to  each  of  the  seventeen  individuals  in  three  “waves”.  No  individual  or  team 
will  possess  sufficient  organic  information  to  accomplish  the  task.  Only  by  sharing  information  (factoids)  can 
the  task  be  accomplished.  Factoid  inter-dependence  will  be  maintained,  i.e.,  a  factoid  can  be  relevant  to  more 
than  one  aspect  of  the  solution  (e.g.  “who”  and  “what”). 

d)  Task:  The  task  of  the  participants  (individual,  team  and  collective)  are  to  identify  the  “who”,  “what”,  “where” 
and  “when”  of  the  attack  within  a  specific  timeframe. 

e)  Information  Sharing  Capabilities:  Individuals  can  send  factoids  to  other  individuals  (share  action),  post 
factoids  to  one  or  more  websites  (post  action)  and  pull  factoids  from  one  or  more  websites  (pull  action). 
However,  permissions  will  be  manipulated  per  C2  Approach  (see  next  section). 

f)  Collaborative  Capabilities:  For  selected  runs,  individuals  will  have  the  ability  to  provide  their  “assessment” 
of  importance  (relevance)  and/or  trustworthiness  of  a  factoid  they  share  or  post. 

g)  Resource  Contention:  The  resources  of  interest  in  ELICIT  are  information  and  cognitive.  Since  information 
that  is  needed  by  one  individual  (team)  can  be  in  the  possession  of  another  one,  a  failure  to  share  or  provide 
access  to  this  information  can  be  considered  as  a  conflict.  Cognitive  efforts  (information  processing  and 
problem  solving)  are  also  required.  There  is  a  limited  amount  of  this  resource  available  and  it  must  be 
allocated  to  the  individual  parts  of  the  problem  (e.g.  who,  what).  Allocating  less  than  sufficient  cognitive 
resources  to  a  part  of  the  problem  can  also  be  considered  a  conflict. 

Experimentation  detailed  design  is  presented  next.  Text  in  italic  is  quoted  from  (Alberts  and  Moffat  2007). 


Model  1:  Conflicted 
COLLECTIVE  OBJECTIVE:  None. 

COLLECTIVE  C2:  None.  The  only  C2  that  exists  is  that  exercised  by  the  individual  entities  over  their  own  forces  or 
organisations. 

C2  IMPLICATIONS: 

•  Each  entity  is  pursuing  its  individual  intent  and  taking  independent  action. 

•  Entities  are  operating  in  the  area  of  operations  without  communicating  with,  sharing  information  with  or 
engaging  in  any  C2-related  interaction.  This  means  that  there  is  no  way  to  avoid  some  ‘ negative  cross¬ 
impact  ’  between  or  among  force  elements. 

ELICIT  SETUP 


Individuals  and  Teams  are  organized  as  follows: 

•  Four  teams  and  one  Coordinator  (isolated). 

•  Each  team  has  four  members  with  one  Team  leader. 
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•  No  team  external  communications  are  allowed. 

•  Coordinator  is  isolated. 


Figure  22  -  ELICIT  Configuration  for  Conflicted  Collective  C2  Approach 


Each  team  operates  independently  its  problem  space  dimension,  namely: 

•  The  Who  team  works  in  the  Who  problem  space. 

•  The  What  team  works  in  the  What  problem  space. 

•  The  Where  team  works  in  the  Where  problem  space. 

•  The  When  team  works  in  the  When  problem  space. 

Overall  Coordinator  (isolated)  works  in  all  problem  space  dimensions. 

The  following  interactions  are  allowed: 

•  Members  can  share  Factoids  within  their  team. 

•  Members  can  post  and  pull  Factoids  from  their  team  website. 

The  following  interactions  are  not  allowed: 

•  Members  cannot  share  Factoids  with  members  outside  their  team. 

•  Members  cannot  post  and  pull  Factoids  from  others  team’s  websites. 

•  (Isolated)  Coordinator  cannot  share  Factoids  with  teams  and  cannot  post  and  pull  Factoids  from  team’s 
websites. 

SUCESS  CRITERION 


Each  Team  pursues  independent  goals.  Therefore,  success  occurs  if  each  Team  leader  finds  the  correct  solution  to  his 
problem  space. 

For  analysis  purposes  (ONLY),  overall  organization  success  will  be  determined  as  all  teams  finding  the  correct 
solution. 

EXPECTED  RESULTS 


The  mission  will  never  be  accomplished  due  to  insufficient  levels  of  information  quality  and  shared  awareness  among 
teams  and  individuals.  Each  team  will  form  isolated  clusters  since  there’ll  be  no  cross-team  interactions. 
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Model  2:  De-conflicted 

COLLECTIVE  OBJECTIVES:  The  avoidance  of  adverse  cross-impacts  between  and  among  the  participants  by 

partitioning  the  problem  space. 

COLLECTIVE  C2:  Limited  to  identifying  potential  conflicts  and  agreements  to  deconflict. 

C2  IMPLICATIONS: 

•  Entities  [...]  must  be  willing,  at  a  minimum  to  accept  a  constraint  on  their  plans  or  actions  (share  information 
across  teams).  In  return  they  hope  to  avoid  or  remove  any  adverse  cross-impacts. 

•  Limited  peer  to  peer  interaction  in  the  Information  domain  must  be  sufficient  to  dynamically  resolve  potential 
cross-impacts. 

•  The  main  emphasis  of  C2  interactions  and  information  flows  is  still  on  vertical  interaction  along  ‘ stove-piped ’ 
chains  of  command  within  each  entity. 


ELICIT  SETUP 


Individuals  and  Teams  are  organized  as  follows: 

•  Four  teams  and  one  Deconflictor. 

•  Each  team  has  four  Team  members  with  one  Team  leader. 

•  Team  leaders  may  distribute  relevant  information  to  Deconflictor. 

•  Deconflictor  role  is  to  distribute  relevant  information  to  Team  leaders. 


ELICIT  Configuration  for  De-conflicted  C2  Approach 

-  Instructions  as  per  ELICIT  Hierarchy  Baseline 

Deconflictor 


Who  Team 


Legend: 

( Deconflictor 
O  Team  leader 
O  Team  member 


Who 

What 

When 

Where 

Web  Site 

Web  Site 

Web  Site 

Web  Site 

Figure  23  -  ELICIT  Configuration  for  De-conflicted  Collective  C2  Approach 


Each  team  operates  independently  its  problem  space  dimension,  namely: 

•  The  Who  team  works  in  the  Who  problem  space. 

•  The  What  team  works  in  the  What  problem  space. 

•  The  Where  team  works  in  the  Where  problem  space. 

•  The  When  team  works  in  the  When  problem  space. 

The  Deconflictor  role  doesn’t  involve  work  in  any  of  the  problem  space  dimensions. 

The  following  interactions  are  allowed: 

•  Deconflictor  can  share  Factoids  with  Team  leaders. 

•  Team  leaders  can  share  Factoids  with  Deconflictor. 
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•  Team  members  can  share  Factoids  within  the  team. 

•  Team  members  can  post  and  pull  Factoids  from  their  team  website. 

The  following  interactions  are  not  allowed: 

•  Deconflictor  cannot  share  Factoids  with  Team  members. 

•  Team  members  cannot  share  Factoids  outside  their  team  (inch  Deconflictor). 

•  Members  cannot  post  and  pull  Factoids  from  others  team’s  websites. 

•  Deconflictor  cannot  access  (post  and  pull  Factoids)  to  websites. 


SUCESS  CRITERION 


Each  Team  pursues  independent  goals.  Therefore,  success  occurs  if  each  Team  leader  finds  the  correct  solution  to  his 
problem  space. 

For  analysis  purposes  (ONLY),  overall  organization  success  will  be  determined  as  all  teams  finding  the  correct 
solution. 

EXPECTED  RESULTS 


There  will  be  a  limited  amount  of  information  sharing  leading  to  a  limited  amount  of  shared  awareness.  Some 
individuals  will  solve  their  part  of  the  problem,  but  few  (if  any)  will  solve  the  entire  problem. 

Strong  formation  of  clusters  within  teams  is  expected.  Active  interactions  are  expected  between  DI  and  Team  leaders 
(stove-pipes  with  centralized  responsibility  and  decision  making). 


Model  3:  Coordinated 

COLLECTIVE  OBJECTIVES:  To  increase  overall  effectiveness  by  1)  seeking  mutual  support  for  intent,  2) 
developing  relationships  and  links  between  and  among  entity  plans  and  actions  to  reinforce  or  enhance  effects  3) 
some  initial  pooling  of  non-organic  resources,  and  4)  increased  sharing  in  the  information  domain  to  increase  the 
quality  of  information. 

COLLECTIVE  C2:  Seeking  opportunities  to  generate  synergy  by  linking  the  plans  and  action(s)  of  one  entity  with 
those  of  another.  Combine  resources  to  achieve  a  necessary  threshold  for  effective  action  or  significant  effects. 

C2  IMPLICATIONS:  This  level  of  maturity  begins  to  make  it  possible  to  form  utask  organised'  forces  with 
contributions  from  different  entities  to  simplify  interactions  across  the  air,  land  and  maritime  domains,  and  other  non¬ 
military  actors. 


ELICIT  SETUP 


Individuals  and  Teams  are  organized  as  follows: 

•  Four  teams  and  one  Coordinator. 

•  Each  team  has  four  Team  members  with  one  Team  leader. 

•  Coordinator  interacts  with  Team  leaders  to  provide  necessary  information  and  orientation  to  determine  the 
correct  solution. 

•  Team  leaders  assist  Coordinator  to  determine  the  correct  solution. 
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Figure  24  -  ELICIT  Configuration  for  Coordinated  Collective  C2  Approach 


Each  team  operates  independently  its  problem  space  dimension,  namely: 

•  The  Who  team  works  in  the  Who  problem  space. 

•  The  What  team  works  in  the  What  problem  space. 

•  The  Where  team  works  in  the  Where  problem  space. 

•  The  When  team  works  in  the  When  problem  space. 

The  Coordinator  works  in  all  problem  space  dimensions. 

The  following  interactions  are  allowed: 

•  Coordinator  can  post  and  pull  Factoids  to  websites. 

•  Coordinator  can  share  Factoids  with  Team  leaders. 

•  Team  leaders  can  share  Factoids  with  Coordinator. 

•  Team  members  can  share  Factoids  within  the  team. 

•  Team  leaders  and  Team  members  can  post  and  pull  Factoids  from  their  team  website. 

The  following  interactions  are  not  allowed: 

•  Coordinator  cannot  share  Factoids  with  Team  members. 

•  Team  members  cannot  share  Factoids  outside  their  team. 

•  Team  leaders  and  members  cannot  post  and  pull  Factoids  from  others  team’s  websites. 


SUCESS  CRITERION 


Organization  success  depends  on  the  Coordinator  finding  the  correct  solution. 
EXPECTED  RESULTS 


There  will  be  increased  information  sharing  leading  to  increased  shared  awareness.  More  individuals  will  solve  their 
part  of  the  problem  and  they  will  solve  it  sooner.  More  individuals  will  solve  the  entire  problem  and  an  overall 
solution  will  be  available  sooner. 

However,  clustering  formation  within  teams  is  still  expected,  with  frequent  interactions  between  Coordinator  and 
Team  leaders  (stove-pipes  with  centralized  responsibility  and  decision  making). 
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Model  4:  Collaborative 

COLLECTIVE  OBJECTIVES:  To  develop  significant  synergies  by  1)  negotiating  and  establishing  shared  intent  and 
a  shared  plan,  2)  establishing  or  reconfiguring  roles,  3)  coupling  actions;  4)  rich  sharing  of  non-organic  resources21 , 
5)  some  pooling  of  organic22  resources;  and  6)  increasing  interactions  in  the  cognitive  domain  to  increase  shared 
awareness. 

COLLECTIVE  C2:  A  collaborative  mechanism  at  the  collective  level  needs  to  be  established  to  develop  a  single 
integrated  plan. 

C2  IMPLICATIONS: 

•  Sharing  of  resources  in  addition  to  a  requirement  for  more  information  sharing  and  interactions  between  and 
among  the  entities. 

•  Richer  peer  to  peer  interworking.  To  a  far  greater  extent  than  is  present  in  lower  levels  of  C2  maturity, 
entities  become  interdependent.  As  a  consequence,  risk  is  pooled  (like  insurance). 

•  This  C2  approach  is  appropriate  for  problems  that  are  not  fully  decomposable  in  terms  of  objectives,  space, 
time,  and  function  and  thus  for  which,  a  holistic  approach  is  desirable. 


ELICIT  SETUP 


Individuals  and  Teams  are  organized  as  follows: 

•  Four  teams  and  one  Coordinator/Facilitator. 

•  Each  team  has  four  Team  members  with  one  Team  leader. 

•  Coordinator/Facilitator  collaborates  with  Team  leaders  and  members  to  provide  necessary  information  and 
knowledge  to  determine  the  correct  solution. 

•  Team  members  may  collaborate  with  members  external  to  own-team  (including  Coordinator)  to  provide 
necessary  information  and  knowledge  to  determine  the  correct  solution. 


ELICIT  Configuration  for  Collaborative  C2  Approach 

Players  have  access  to  all  websites 

Feature  added  that  allows  players  to  evaluate 

factoids  and  share  /post  evaluations  Coordinator/ 

Instructions  define  role  of  coordinator  Facilitator 


Who  Team 


Legend: 

Coordinator 
O  Team  leader 
O  Team  member 


Figure  25  -  ELICIT  Configuration  for  Collaborative  Collective  C2  Approach 


Each  team  operates  independently  its  problem  space  dimension,  namely: 

•  The  Who  team  works  in  the  Who  problem  space. 

•  The  What  team  works  in  the  What  problem  space. 

•  The  Where  team  works  in  the  Where  problem  space. 


21  Non-organic  resources  refers  to  resources  not  ‘owned’  by  participants. 

22  Organic  resources  are  those  ‘owned’  by  a  participant. 
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•  The  When  team  works  in  the  When  problem  space. 

The  Coordinator/Facilitator  works  in  all  problem  spaces. 

The  following  interactions  are  allowed: 

•  Coordinator  can  post  and  pull  Factoids  to  websites. 

•  Coordinator  can  share  Factoids  with  Team  leaders  and  members,  and  vice-versa. 

•  Team  members  can  share  Factoids  within  members  external  to  own-team. 

•  Team  members  can  post  and  pull  Factoids  from  websites. 

The  following  interactions  are  not  allowed: 

•  None. 

Additionally,  the  following  information  can  be  attached  to  each  Factoid  share/post: 

•  Relevance; 

•  Trustworthiness. 

SUCESS  CRITERION 


Organization  success  depends  on  the  Coordinator  finding  the  correct  solution  to  all  problem  spaces  AND  Team 
leaders  finding  the  correct  solution  to  their  problem  space. 

EXPECTED  RESULTS 


There  will  be  significantly  increased  information  sharing  including  significant  amounts  of  reciprocity  (sharing 
evaluations  with  those  who  share  evaluated  factoids)  leading  to  significantly  increased  shared  awareness.  More  (than 
at  the  coordinated  C2  level)  members  will  solve  their  part  of  the  problem  and  they  will  solve  it  sooner.  More 
members  will  solve  the  entire  problem  and  an  overall  solution  will  be  available  sooner. 

Less  formation  of  team-based  clusters  with  increased  and  richer 23  peer-to-peer  interactions  across  members,  leaders 
and  coordinator.  No  stove-pipes,  but  some  centralized  responsibility  and  decision  making  (e.g.,  valid  indentifles). 


Model  5:  Edge 

COLLECTIVE  OBJECTIVES:  To  provide  the  enterprise  with  additional  C2  approach  options  that  involve  entities 
working  more  closely  together  and  with  the  ability  to  identify  and  implement  the  most  appropriate  approach  to 
coalition  C2  given  the  situation  (e.g.,  mission,  conditions,  and  set  of  coalition  partners/contributing  entities). 

COLLECTIVE  C2:  Emergent  delegations  of  decision  rights.  These  delegations  are  both  dynamic  and  tailored  as  a 
function  of  the  entities  and  the  situation 

C2  IMPLICATIONS: 

•  Need  for  a  high  degree  of  shared  understanding  of  a  common  (collective)  intent. 

•  Need  for  rich  and  continuous  set  of  interactions  between  /  among  participants,  involving  widespread 
information  exchanges  to  allow  the  build  up  of  shared  understanding,  and  the  ability  (where  appropriate)  to 
self-synchronize 


ELICIT  SETUP 


Flat  organization  with  17  individuals  (or  entities).  Individuals  can  collaborate  and  exchange  information  (e.g., 
Factoids  with  evaluation  data)  to  provide  necessary  information  and  knowledge  to  determine  the  correct  solution. 


23  Interactions  are  richer  due  to  added  feature  that  allows  subjects  to  evaluate  factoids  in  terms  of  relevance  and  trustworthiness  in  each  share  and  post  action. 
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ELICIT  Configuration  for  Edge  C2  Approach 


Figure  26  -  ELICIT  Configuration  for  Edge  Collective  C2  Approach 


Individuals  work  in  all  problem  space  dimensions  (i.e.,  who ,  what ,  where  and  when). 

The  following  interactions  are  allowed: 

•  Individuals  can  post  and  pull  Factoids  to  websites. 

•  Individuals  can  share  Factoids  with  other  individuals. 

The  following  interactions  are  not  allowed: 

•  None. 

Additionally,  the  following  information  can  be  attached  to  each  Factoid  share/post: 

•  Relevance; 

•  Trustworthiness. 

SUCESS  CRITERION 


Organization  success  depends  on  the  majority  of  Individuals  finding  the  correct  solution  to  all  problem  spaces. 
EXPECTED  RESULTS 

There  will  be  dramatically  increased  information  sharing  (more  than  at  the  collaborative  level)  including  significant 
amounts  of  reciprocity  (posting  evaluations  on  websites  that  have  evaluations  already  posted  and  sharing  evaluations 
with  those  from  whom  evaluated  factoids  were  received)  leading  to  dramatically  increased  shared  awareness.  More 
(than  at  the  collaborative  C2  approach)  individuals  will  solve  the  complete  problem  and  they  will  solve  it  sooner. 

No  clusters  will  be  discernible  (or  will  be  ad-hoc,  without  prior  engagement)  with  increased  and  richer  (i.e.,  Factoids 
evaluation)  peer-to-peer  interactions  across  individuals.  No  stove-pipes,  with  decentralized  responsibility  and  decision 
making. 
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MODEL  INTERVENING  VARIABLES 

The  experimentation  model  included  relevant  intervening  variables,  labeled  as  enablers/inhibitors.  Their  control  or 
manipulation  is  out  of  the  scope  of  this  work  (except  for  Task  Difficulty).  Nonetheless,  they  are  described  next,  in 
Table  16,  which  may  be  used  as  reference  for  future  related  research. 


Name 

Description 

Individual  and  Team 
Characteristics 

These  encompass  the  human  complexity  aspects  which  we  cannot  control  (e.g.,  IQ, 
memory,  language  understanding,  idiosyncrasies,  (natural)  propensity  to  share), 
although  some  we  may  influence.  An  example  is  sharing  behavior  and 
collaboration,  which  may  be  enabled  (or  inhibited)  by  setting  collective  (or 
isolated)  goals. 

The  following  variables  of  interest  will  be  monitored/observed: 

-  Culture  and  Social24. 

-  Team  Hardness  (SAS-050  2007):  as  a  run  progresses25,  new  relations  and  links 
emerge.  This  is  observable  in  ELICIT  by,  for  example,  the  analysis  of  social  ties 
(strong  and  weak)  and,  if  applicable,  the  formation  of  clusters. 

-  Trust  (SAS-050  2007):  the  provided  environment  does  not  influence  individuals’ 
trust  levels  (e.g.,  use  of  anonymity,  no  wrong  information,  all  necessary 
information  available,  no  instructions  to  hoard  and  no  antagonistic  objectives)26. 

-  Training :  Training  is  provided  to  individuals  by  means  of  a  trial  run  before 
conducting  the  exercise.  The  trial  presents  the  same  environment  than  the  actual 
exercise,  providing  sufficient  learning  on  how  to  interact  with  the  platform. 

-  Experience :  We  consider  existing  professional  past  experience  will  not  play  a 
major  role  in  the  ELICIT  game27. 

-Intelligence  Quotient  (see  Individual  Cognitive  Abilities,  SAS-050):  Students 
from  the  military  academy  (equivalent  to  university)  will  be  used  as  subjects  for 
the  ELICIT  experiments.  Therefore,  all  individuals  should  have  sufficient  IQ  levels 
to  conduct  the  experiment  (i.e.,  understand  factoids  semantically,  discern  those 
which  are  relevant  and  share/post  appropriately). 

See  also  Information  Sharing  and  Collaboration. 

Task  Difficulty 

Currently,  there  are  four  ELICIT  problems  (i.e.,  four  factoids  sets):  all  have  the 
same  difficulty  level  except  set  2  which  is  more  difficult28. 

-  Primary  goal  of  the  experiment  is  to  observe  that,  for  problems  having  the  same 
difficulty  level,  more  mature  organizations  outperform  less  mature  ones  (i.e.,  they 
are  more  effective  and  more  efficient  -  better  Task  Performance). 

-  Secondary  goal  of  the  experiment  is  to  observe  that,  for  increased  problem 
difficulty,  more  mature  organizations  are  more  effective. 

Quality  of  Information  sources 

Currently,  factoids  are  not  tagged  or  evaluated  based  on  its  source.  This  variable  is 
currently  fixed,  although  it  may  be  manipulated  for  future  experiments. 

Quality  of  (Communications) 
Infrastructure 

The  communications  infrastructure  has  sufficient  capacity  and  performance  to 
share  and  receive  information  (fixed  value). 

Platform 

ELICIT  software  platform  is  also  an  important  factor  in  the  experiment:  the  way 
information  is  presented,  the  level  interactions  allowed  between  subjects  (e.g., 
knowledge  sharing)  and  functionalities  (e.g.,  collaborative  boards)  are  aspects 
which  may  be  introduced  or  enhanced  with  influence  on  several  network  variables 
(e.g.,  information  distribution,  patterns  of  interactions,  shared  understanding). 

Table  16  -  Model  intervening  variables 


24  Recent  research  by  nGenera  (2009,  Tapscott),  based  on  more  than  1 1  000  surveys,  suggests  that  habits,  skills  and  traits  of  NetGen,  a  generation  bom  between 
1977  and  1997,  are  significantly  different  than  those  generations  before.  Tapscott  identified  NetGet  main  differentiating  characteristics,  when  compared  with 
previous  generations,  as:  need  for  freedom,  need  to  customize  and  personalize,  scmtinizers,  integrity  and  open,  wants  entertainment  and  play,  collaborative  (social 
networks),  want  speed,  and  innovative.  Inherent  to  this  generation  is  a  native  understanding  and  operation  of  new  technologies  and  internet  2.0.  NetGen  people  are 
starting  to  enter  the  work  force,  including  military,  and  their  open,  collaborative  and  social  practices  clash  with  industrial  age  practices. 

25  Variable  time  is  implicit  here. 

26  See  (Powley  and  Nissen  2009)  for  related  ELICIT  experimentation  on  tmst  (and  mistmst). 

27  Team  hardness  optional  package  will  specifically  address  this  issue. 

28  ‘What’  solution  for  factoid  set  2  has  an  increased  option  space  than  all  other  sets.  Specifically,  while  all  other  sets  present  over  4  possibilities,  set  2  allows  8 

possibilities  (e.g.,  attack  will  be  at  embassy  of  a  coalition  countries,  which  comprises  5  countries). _ 
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DATA  TABLES  AND  CHARTS 

Information  Domain  Scores 


Shared 

Info.  Accessible 

RELEVANT 

Shared 

Info.  Accessible 

Shared 

Info.  Reached 

RELEVANT 

Shared 

Info.  Reached 

Run  ID 

BEST  % 

BEST  %  at  time 
(sec) 

BEST  % 

BEST  %  at  time 

(sec) 

BEST  % 

BEST  %  at  time 

(sec) 

BEST  % 

BEST  %  at  time 

(sec) 

LI-01 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

LI -02 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

LI -03* 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L2-01 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L2-02 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L2-03 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L2-04 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L3-01 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L3-02 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L3-03* 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L3-04* 

0% 

- 

0% 

- 

0% 

- 

0% 

- 

L4-01 

65% 

1  351 

65% 

722 

65% 

1  855 

65% 

1  687 

L4-02 

80% 

702 

80% 

684 

30% 

947 

50% 

1  432 

L4-03 

50% 

622 

65% 

2  303 

30% 

1  656 

30% 

1  656 

L4-04* 

65% 

234 

65% 

218 

20% 

1  626 

20% 

1  626 

L5-01 

65% 

2  219 

65% 

677 

50% 

979 

65% 

1  139 

L5-02 

80% 

1  304 

65% 

664 

65% 

1  093 

65% 

999 

L5-03 

80% 

1  172 

80% 

652 

0% 

0 

0% 

0 

*  Early  delivery  of  factoids  from  server  as  illustrated  in  Figure  3 

Table  17  -  Information  Domain  Scores:  Shared  Information 
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Information  Reached 

CTC 

TL  WHO 

TL  WHAT 

TL  WHERE 

TL  WHEN 

AVG  TLs 

Run  ID 

Best  % 

BEST  %  at  time  (sec) 

Best  % 

BEST  %  at  time  (sec) 

Best  % 

BEST  %  at  time  (sec) 

Best  % 

BEST  %  at  time  (sec) 

Best  % 

BEST  %  at  time  (sec) 

Best  % 

LI -01 

0% 

20% 

581 

20% 

473 

20% 

1869 

10% 

600 

18% 

LI -02 

0% 

- 

10% 

378 

20% 

1049 

10% 

175 

10% 

430 

13% 

LI -03* 

0% 

- 

20% 

520 

10% 

288 

10% 

86 

20% 

601 

15% 

L2-01 

30% 

1222 

30% 

1749 

20% 

624 

20% 

626 

20% 

928 

23% 

L2-02 

50% 

1074 

20% 

1385 

20% 

311 

20% 

600 

30% 

1440 

23% 

L2-03 

30% 

1113 

50% 

1964 

30% 

600 

30% 

612 

50% 

1362 

40% 

L2-04 

30% 

1016 

30% 

1431 

30% 

809 

30% 

1620 

30% 

848 

30% 

L3-01 

65% 

1049 

20% 

412 

20% 

629 

20% 

751 

20% 

1857 

20% 

L3-02 

80% 

735 

20% 

758 

30% 

1737 

20% 

775 

20% 

553 

23% 

L3-03* 

80% 

1420 

20% 

466 

20% 

304 

20% 

303 

10% 

128 

18% 

L3-04* 

80% 

1476 

20% 

342 

20% 

337 

20% 

1448 

20% 

1385 

20% 

L4-01 

65% 

838 

80% 

1453 

65% 

987 

65% 

1045 

65% 

913 

69% 

L4-02 

80% 

844 

80% 

866 

80% 

868 

100% 

976 

50% 

709 

78% 

L4-03 

65% 

984 

50% 

1496 

65% 

860 

65% 

1653 

65% 

1068 

61% 

L4-04* 

80% 

1608 

50% 

1016 

30% 

907 

80% 

1651 

65% 

486 

56% 

L5-01 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

L5-02 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

L5-03 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

*  Early  delivery  of  factoids  from  server  as  illustrated  in  Figure  3 

Table  18  -  Information  Domain  Scores:  Information  Reached  and  Reach  per  role 
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Information  Reach  and  Information  Reached  by  Subjects 

Examples  of  Information  Reach  by  Subjects  per  approach  is  presented  in  next  tables.  Colors  are  used  to  facilitate 
awareness  of  factoids  relevance:  K/E,  S  and  N  are  colored  Green,  Orange  and  Grey  respectively;  WHITE  cells  means 
no  subject  access  to  factoid. 


Table  19  -  EDGE  run  L5-02:  Information  Reach  (top)  and  Information  Reached  (bottom)  by  Subjects 


Table  20  -  EDGE  run  L5-03:  Information  Reach  (top)  and  Information  Reached  (bottom)  by  Subjects 
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Table  21  -  COLLABORATIVE  run  L4-01:  Information  Reach  (top)  and  Information  Reached  (bottom)  by  Subjects 


Table  22  -  COLLABORATIVE  run  L4-02:  Information  Reach  (top)  and  Information  Reached  (bottom)  by  Subjects 
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Table  24  -  DECONFLICTED  run  L2-03:  Information  Reach  (top)  and  Information  Reached  (bottom)  by  Subjects 


Table  25  -  CONFLICTED  run  Ll-01:  Information  Reach  by  Subjects 
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Nature  of  Interactions 


Run  ID 

LEVEL 

Shares  per  Hour 

Posts  per  Hour 

Pulls  per  Hour 

Ids  per  Hour 

OVERALL 
per  Hour 

LI-01 

LI 

264 

133 

164 

65 

625 

LI -02 

LI 

129 

88 

143 

47 

407 

LI -03 

LI 

118 

188 

203 

45 

553 

L2-01 

L2 

309 

135 

242 

63 

749 

L2-02 

L2 

264 

190 

156 

65 

675 

L2-03 

L2 

363 

174 

109 

49 

695 

L2-04 

L2 

455 

217 

145 

42 

858 

L3-01 

L3 

285 

91 

235 

40 

651 

L3-02 

L3 

321 

211 

161 

70 

763 

L3-03 

L3 

141 

121 

140 

49 

451 

L3-04 

L3 

237 

128 

208 

73 

645 

L4-01 

L4 

102 

161 

729 

71 

1  062 

L4-02 

L4 

150 

272 

655 

101 

1  177 

L4-03 

L4 

95 

137 

654 

66 

951 

L4-04 

L4 

168 

372 

667 

51 

1  258 

L5-01 

L5 

137 

140 

1  443 

91 

1  811 

L5-02 

L5 

102 

187 

1  055 

105 

1  449 

L5-03 

L5 

69 

180 

1  082 

60 

1  391 

Table  26  -  Nature  of  Interactions:  totals  per  hour 


Value 

APPROACH  \ 

Shares  per  Hour 

Posts  per  Hour 

Pulls  per  Hour 

Ids  per  Hour 

OVERALL 
per  Hour 

CONFLICTED 

176 

131 

167 

53 

527 

DECONFLICTED 

350 

178 

164 

54 

746 

COORDINATED 

240 

138 

183 

58 

619 

COLLABORATIVE 

126 

228 

677 

72 

1  104 

EDGE 

105 

167 

1  209 

86 

1  567 

Table  27  -  Nature  of  Interactions:  average  of  totals  per  hour  per  approach 
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In  and  Out  Flows:  Deviation  from  Average 


CTC/TLs 

TMs 

APPROACH 

IN 

STD  DEV 

OUT 

STD  DEV 

IN 

STD  DEV 

OUT 

STD  DEV 

CONFLICTED 

4.1 

4.8 

-0.4 

3.0 

-0.5 

4.1 

0.9 

4.7 

DECONFLICTED 

16.8 

12.0 

15.5 

23.3 

-7.0 

6.9 

-6.5 

10.7 

COORDINATED 

12.4 

19.3 

3.0 

6.5 

-5.1 

5.4 

-1.2 

8.1 

COLLABORATIVE 

3.0 

12.1 

-4.2 

5.3 

-1.2 

11.3 

1.7 

11.0 

EDGE 

- 

- 

- 

- 

0.0 

19.1 

0.0 

9.0 

Table  28  -  Node  Balance  per  Role 


RUN  ID 

Node  Balance 
(IN) 

Node  Balance 
(OUT) 

IN  AVG 

Coord  and  TLs 

IN  AVG 

TMs 

OUT  AVG  Coord 
and  TLs 

OUT  AVG 

TMs 

LI-01 

5.2 

-0.6 

-0.3 

1.2 

LI -02 

2.0 

0.1 

-0.3 

0.7 

LI-03 

5.0 

-1.0 

-0.5 

0.7 

L2-01 

10.6 

-4.4 

5.6 

-2.3 

L2-02 

10.8 

-4.5 

6.2 

-2.6 

L2-03 

27.0 

-11.3 

26.6 

-11.1 

L2-04 

18.7 

-7.7 

23.7 

-9.9 

L3-01 

13.4 

-5.6 

6.2 

-2.6 

L3-02 

13.7 

-5.7 

-0.6 

0.3 

L3-03 

14.8 

-6.2 

5.6 

-2.3 

L3-04 

7.7 

-3.2 

0.6 

-0.2 

L4-01 

9.8 

-4.1 

-1.7 

0.7 

L4-02 

1.8 

-0.7 

-7.1 

2.9 

L4-03 

0.2 

-0.1 

-0.9 

0.3 

L4-04 

0.0 

0.0 

-7.0 

2.9 

L5-01 

- 

- 

- 

- 

L5-02 

- 

- 

- 

- 

L5-03 

- 

- 

- 

- 

Table  29  -  Node  Balance  per  Role 
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Sociograms 


L5-02  Sociogram  (no  websites) 


L4-01  Sociogram  (no  websites) 


(10  or  more  interactions,  incl  websites) 


(10  or  more  interactions,  incl  websites) 


£Mcrg.-ui 

©WHO 

(10  or  more  interactions,  incl  websites) 


(10  or  more  interactions,  incl  websites) 

©WHERE 

©Whey  ©Tayfe* 


pQurtfi  ©  idle 

©Pa*  ©Chra 


©WHAT 


©  M  WDJfi  ©Lnir 


Neds  Balance 


I 

1 

I 

\ 

1 


Ll-02  Sociogram  (no  websites) 


(10  or  more  interactions,  incl  websites) 


Figure  27  -  Sociograms  (first  2  columns)  and  IN-OUT  Deviation  per  role  (last  column) 
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Quality  of  Interactions 


Quality  of 
Interactions 
(Shares  and 
Posts) 

Quality  of  Interactions 
(Shares  and  Posts) 

Log  File 

LEVEL 

Factoid  Set 

Trial 

TOTAL  Value 

TOTAL  Value  / 
Subject 

Tendency 

CTC 

TLs  (avg) 

WHO  TL 

WHAT  TL 

WHERE  TL 

WHEN  TL 

Avg  TMs 

LI-01 

D 

4 

B 

2.00 

0.12 

- 

0.00 

1.50 

2.00 

1.00 

5.00 

-2.00 

LI -02 

1 

10.00 

0.59 

+ 

0.00 

2.50 

1.00 

3.00 

1.00 

5.00 

LI -03 

D 

3 

B 

8.00 

0.47 

+ 

0.00 

1.25 

2.00 

1.00 

-2.00 

4.00 

L2-01 

L2 

■ 

N 

-8.00 

-0.47 

■ 

-9.00 

-3.00 

-2.00 

-10.00 

2.00 

-2.00 

■ 

L2-02 

B 

4 

B 

-1.00 

-0.06 

= 

5.00 

0.50 

-1.00 

-5.00 

0.00 

8.00 

L2-03 

1 

52.00 

3.06 

+ 

21.00 

6.00 

-1.00 

8.00 

14.00 

3.00 

L2-04 

3 

-15.00 

-0.88 

- 

-13.00 

-3.50 

-15.00 

0.00 

-20.00 

21.00 

L3-01 

1 

11.00 

0.65 

+ 

-4.00 

0.00 

7.00 

-7.00 

2.00 

-2.00 

L3-02 

3 

18.00 

1.06 

+ 

11.00 

1.75 

3.00 

2.00 

-2.00 

4.00 

L3-03 

2 

26.00 

1.53 

+ 

6.00 

1.50 

2.00 

4.00 

-3.00 

3.00 

L3-04 

4 

3.00 

0.18 

- 

-2.00 

-0.25 

2.00 

4.00 

-7.00 

0.00 

L4-01 

B 

1 

15.00 

0.88 

+ 

1.00 

-0.50 

-3.00 

2.00 

1.00 

-2.00 

L4-02 

L4 

4 

63.00 

3.71 

+ 

1.00 

1.50 

3.00 

0.00 

2.00 

1.00 

L4-03 

B 

3 

72.00 

4.24 

+ 

3.00 

3.00 

0.00 

3.00 

-1.00 

10.00 

L4-04 

L4 

2 

30.00 

1.76 

+ 

-2.00 

3.75 

7.00 

3.00 

0.00 

5.00 

L5-01 

B 

1 

10.00 

0.59 

ii 

0.59 

L5-02 

4 

51.00 

3.00 

+ 

3.00 

L5-03 

B 

4 

B 

19.00 

1.12 

+ 

1.12 

Table  30  -  Quality  of  Interactions  per  run  per  role 


APPROACH 

AVG 

MIN 

MAX 

CONFLICTED 

6.67 

2.00 

10.00 

DECONFLICTED 

7.00 

-15.00 

52.00 

COORDINATED 

11.63 

0.17 

26.00 

COLLABORATIVE 

45.00 

15.00 

72.00 

EDGE 

26.67 

10.00 

51.00 

Table  31  -  Quality  of  Interactions:  mean  value  per  approach 
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Cognitive  Self-Synchronization 


Self-Synchronization  (cognitive) 


|—  overall] 

Ll-02  Run 


Self-Synchronization  (cognitive) 


|— overall] 

L2-03  Run 


Self-Synchronization  (cognitive) 

Self-Synchronization  (cognitive) 

1,00 

0.95 

0.95 

0.90 

0.90 

0.85 

0,85 

0.80 

0.80 

0.75 

0.75 

0.70 

^  0.70 

IT  0.65 

<1) 

ii  0,65 
c 

|o.eo 

o  0,60 

'c  0,55 

l  °'55 

B  0.50 

§  0.50 

2  0.46 

E  0.45 

!°.40 

c  o,40 

1  0.35 

sL  0,35 
CO 

03* 

ojfeaoqao  | 

0.30 

0,30 

r1 

0.25 

0.25 

T 

0.20 

0.15 

J 

0.20 

0.15 

o  .uflWl 

0.10 

npa  'frjfe'  ooa  -cffir 

0.10 

0.05 

0,05 

zm*  ^  033  °f‘ 

Q.oo  ojaKH— ■ - - 1 

0.00 

( 

)  250  500  750  1.000  1.250  1.500  1.750  2.000  2.250 

D’°°( 

)  250  500  750  1.000  1.250  1.500  1.750  2.000 

|  — OVERALL  | 

|  — OVERALL  | 

L3-04  Run  L4-04  Run 


Self-Synchronization  (cognitive) 


|— OVERALL  | 

L5-02  Run 

Figure  28  -  Example  Charts  of  time  evolution  of  Disorder  of  Cognitive  Self-Synchronization  per  approach 
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Cognitive  Self-Synchronization 

ID 

WHO 

WHAT 

WHERE 

WHEN  (t) 

WHEN  (d) 

WHEN  (m) 

OVERALL 

LI-01 

0.10 

0.09 

0.14 

0.10 

0.00 

0.00 

0.088 

LI -02 

0.10 

0.07 

0.06 

0.17 

0.00 

0.03 

0.071 

LI-03 

0.00 

0.00 

0.00 

0.00 

0.03 

0.03 

0.004 

L2-01 

0.07 

0.03 

0.03 

0.07 

0.00 

0.03 

0.039 

L2-02 

0.00 

0.00 

0.13 

0.17 

0.07 

0.03 

0.052 

L2-03 

0.31 

0.12 

0.28 

0.03 

0.35 

0.22 

0.219 

L2-04 

0.23 

0.17 

0.21 

0.20 

0.07 

0.12 

0.179 

L3-01 

0.06 

0.14 

0.14 

0.07 

0.00 

0.10 

0.097 

L3-02 

0.17 

0.14 

0.23 

0.00 

0.07 

0.07 

0.145 

L3-03 

0.06 

0.10 

0.23 

0.22 

0.12 

0.07 

0.127 

L3-04 

0.25 

0.17 

0.34 

0.20 

0.00 

0.12 

0.215 

L4-01 

0.33 

0.38 

0.42 

0.77 

0.48 

0.55 

0.416 

L4-02 

0.53 

0.38 

0.25 

0.62 

0.20 

0.48 

0.386 

L4-03 

0.18 

0.14 

0.48 

0.12 

0.06 

0.03 

0.217 

L4-04 

0.28 

0.29 

0.39 

0.48 

0.34 

0.41 

0.331 

L5-01 

0.41 

0.33 

0.58 

0.69 

0.55 

0.00 

0.424 

L5-02 

0.35 

0.43 

0.40 

0.84 

0.55 

0.03 

0.401 

L5-03 

0.40 

0.41 

0.32 

0.77 

0.22 

0.55 

0.398 

Table  32  -  Cognitive  Self-Synchronization  per  run 


APPROACH 

Mean 

MIN 

MAX 

CONFLICTED 

0.05 

0.00 

0.09 

DECONFLICTED 

0.12 

0.04 

0.22 

COORDINATED 

0.15 

0.10 

0.22 

COLLABORATIVE 

0.34 

0.22 

0.42 

EDGE 

0.41 

0.40 

0.42 

Table  33  -  Average  value  of  Cognitive  Self-Synchronization  per  approach 
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Cognitive  Domain 


Run  ID 

WHO 

WHAT 

WHERE 

WHEN 

_i 

< 

l- 

O 

l- 

LI-01 

2(17) 

0(17) 

1(17) 

1(17) 

4(64) 

LI -02 

2(17) 

0(17) 

2(17) 

3(17) 

7(64) 

LI -03 

1(17) 

0(17) 

0(17) 

2(17) 

3(64) 

L2-01 

1(17) 

0(17) 

1(17) 

2(17) 

4(64) 

L2-02 

1(17) 

0(17) 

1(17) 

3(17) 

5(64) 

L2-03 

7(17) 

1(17) 

6(17) 

5(17) 

19(64) 

L2-04 

4(17) 

5(17) 

3(17) 

4(17) 

16(64) 

L3-01 

0(17) 

0(17) 

1(17) 

2(17) 

3(64) 

L3-02 

5(17) 

1(17) 

0(17) 

2(17) 

8(64) 

L3-03 

1(17) 

0(17) 

4(17) 

3(17) 

8(64) 

L3-04 

5(17) 

1(17) 

2(17) 

3(17) 

1 1  (64) 

L4-01 

5(17) 

2(17) 

2(17) 

9(17) 

18(64) 

L4-02 

9(17) 

3(17) 

6(17) 

9(17) 

27  (64) 

L4-03 

4(17) 

4(17) 

9(17) 

3(17) 

20  (64) 

L4-04 

6(17) 

0(17) 

5(17) 

8(17) 

19(64) 

L5-01 

8(17) 

1(17) 

3(17) 

5(17) 

17(64) 

L5-02 

4(17) 

4(17) 

7(17) 

9(17) 

24  (64) 

L5-03 

7(17) 

3(17) 

6(17) 

10(17) 

26  (64) 

Table  34  -  Extent  of  Correct  Understanding 
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Overall  Scores  Table 


MoM  -  Effectiveness 
(by  sol  space) 

Time  Effectiveness  (sec) 

Run  ID 

WHO 

WHAT 

WHERE 

WHEN  (time) 

WHEN  (day) 

WHEN  (month) 

WHO 

i- 

< 

X 

£ 

WHERE 

WHEN 

_i 

_i 

2 

LU 

> 

O 

LI-01 

SILVER 

SIGMALAND 

9:00AM 

14 

MAIO 

0.25 

0.00 

0.00 

0.09 

0.34 

1  987 

LI -02 

AZUR 

PSILAND 

PSILAND 

11:00AM 

5 

ABRIL 

0.00 

0.00 

0.25 

0.17 

0.42 

1  519 

LI -03 

IVORY 

TERMINAIS  DE 
GASODUTOS 

15 

AGOSTO 

0.00 

0.00 

0.00 

0.16 

0.16 

259 

L2-01 

VIOLETA 

EMBAIXADA 

TAULAND 

11:00AM 

- 

ABRIL 

0.25 

0.00 

0.00 

0.17 

0.42 

2  376 

L2-02 

BANCO 

PI  LAND 

9:00AM 

- 

JANEIRO 

0.00 

0.00 

0.00 

0.17 

0.17 

1  698 

L2-03 

VIOLETA 

DIGNATARIOS 

PSILAND 

0.25 

0.00 

0.25 

0.00 

0.50 

2  270 

L2-04 

BEIGE 

TERMINAL  DE 
OLEODUTO 

THETALAND 

11:00PM 

14 

AGOSTO 

0.00 

0.25 

0.25 

0.17 

0.67 

2  174 

L3-01 

0.00 

0.00 

0.00 

0.00 

0.00 

- 

L3-02 

MAGENTA 

ESTACAO  DE 
COMBOIOS 

LAMBDALAND 

- 

DIA  SA 
NTO 

- 

0.25 

0.00 

0.00 

0.00 

0.25 

1  509 

L3-03 

JUPITER 

EMBAIXADA 

BETALAND 

3:00PM 

16 

MAIO 

0.00 

0.00 

0.00 

0.17 

0.17 

1  283 

L3-04 

SILVER 

ESCOLA 

SECULAR 

OMICRONLAND 

9:00AM 

- 

JANEIRO 

0.25 

0.25 

0.25 

0.17 

0.92 

1  840 

L4-01 

VIOLETA 

INSTITUICAO 

FINANCEIRA 

PSILAND 

11:00AM 

5 

ABRIL 

0.25 

0.25 

0.25 

0.25 

1.00 

2  028 

L4-02 

SILVER 

ESCOLA 

OMICRONLAND 

9:00AM 

Terga- 

Feira 

JANEIRO 

0.25 

0.00 

0.25 

0.17 

0.67 

1  698 

L4-03 

MAGENTA 

TERMINAL  DE 
OLEODUTO 

THETALAND 

11:00PM 

- 

- 

0.25 

0.25 

0.25 

0.09 

0.84 

2  355 

L4-04 

AQUA 

EMBAIXADA 

ALPHALAND 

3:00PM 

16 

MAIO 

0.25 

0.00 

0.25 

0.17 

0.67 

1  899 

L5-01 

VIOLETA 

EMBAIXADA  DE 
TAULAND 

EPSILONLAND 

11:00AM 

10 

JAN,  JUN 

0.25 

0.00 

0.00 

0.09 

0.34 

1  432 

L5-02 

SILVER 

ESCOLA 

OMICRONLAND 

9:00AM 

1 

JANEIRO 

0.25 

0.00 

0.25 

0.25 

0.75 

1  241 

L5-03 

SILVER 

ESCOLA 

OMICRONLAND 

9:00AM 

1 

JANEIRO 

0.25 

0.00 

0.25 

0.25 

0.75 

1  658 

Table  35  -  Identifications  and  Scores  per  run 


APPROACH 

WHO 

WHAT 

WHERE 

WHEN 

OVERALL 

MIN 

MAX 

CONFLICTED 

0.08 

0.00 

0.08 

0.14 

0.31 

0.16 

0.42 

DECONFLICTED 

0.13 

0.06 

0.13 

0.13 

0.44 

0.17 

0.67 

COORDINATED 

0.13 

0.06 

0.06 

0.09 

0.34 

0.00 

0.92 

COLLABORATIVE 

0.25 

0.13 

0.25 

0.17 

0.80 

0.67 

1.00 

EDGE 

0.25 

0.00 

0.17 

0.20 

0.61 

0.34 

0.75 

Table  36  -Identifications  Scores  Average  per  approach 
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Effort  Spent 


Effort  (cost) 


1800 

1600 

1400 

1200 

1000 

800 

600 

400 

200 

0 


CONFLICTED  DECONFLICTED  COORDINATED  COLLABORATIVE  EDGE 


■  Ids  per  Hour 

■  Pulls  per  Hour 

■  Posts  per  Hour 

■  Shares  per  Hour 


Figure  29  -  Effort  spent  per  approach 
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EXPLANATION  OF  STATISTICAL  CHARTS 


Organiz; 


The  few  number  of  runs  per  approach  available  for 
analysis  is  not  enough  for  a  robust  statistical  analysis. 
Yet,  it  enables  the  identification  of  preliminary  results 
and  trends. 

Charts  displaying  mean  value  in  conjunction  with 
maximum  and  minimum  values  of  the  dataset  were 
frequently  used.  This  method  provides  a  visual 
representation  to  observe  results  range  and  mean  trend 
(towards  maximum  or  minimum). 


EXPLANATION  OF  IN  AND  OUT  FLOWS  CHART 

The  chart  center  (0 , 0)  represents  the  IN  and  OUT  average  per  approach.  Each  node  ( bubble )  represents  a  given  role 
and  approach.  The  more  distant  a  node  is  from  the  center  it  means  its  behavior  is  more  deviant  (from  normal). 
Furthermore,  different  interpretations  are  given  depending  on  the  value  location: 

•  Nodes  located  in  quadrant  1  (right  top  part)  have  propensity  for  in  and  out  flows  (receive  and  share). 

•  Nodes  located  in  quadrant  2  (left  top  part)  have  propensity  for  out  but  not  for  in  flows  (not  receive  and  share). 

•  Nodes  located  in  quadrant  3  (left  bottom  part)  don’t  have  propensity  for  out  nor  for  in  flows  (not  receive  and 
not  share). 

•  Nodes  located  in  quadrant  4  (right  bottom  part)  have  propensity  for  in  but  not  for  out  flows  (receive  and  not 
share). 

•  Nodes  circles  diameters  are  proportional  to  values’  standard  deviation. 

For  clarity: 

•  Coordinator  and  leader  roles  have 
respective  circumference  colored  in  RED 

•  Team  members  have  respective 
circumference  colored  in  GREEN. 

•  Circles  areas  are  colored  according  to 

approach  as  follows:  CONFLICTED  is 
RED,  DECONFLICTED  is  GREY, 
COORDINATED  is  BLUE, 

COLLABORATIVE  is  GREEN. 

•  EDGE  is  not  applicable  (all  subjects  are 
assigned  as  team  members).  In 
CONFLICTED,  isolated  coordinator  was 
excluded  from  calculations. 
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Background 


■  C2  in  the  Information  Age: 

-  Theory  of  NCW,  including  NCW  tenets, 

NCW  Value  Chain  and  C2  Approach  Space  (CCRP, 
Alberts  and  Hayes) 

-  C2  models:  C2  CRM  (SAS-050) 

-  NATO  NEC  C2  Maturity  Model  (SAS-065)  1,  recently 
developed  and  benefiting  from  multiple  validation 
methods,  including  experimentation 


1  NATO  SAS-065.  NATO  NEC  C2  Maturity  Model.  CCRP  Publication  Series,  2010. 
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Background:  N2C2M2 


■  NATO  NEC  C2  Maturity  Model 

-  Defines  5  levels  of  NATO  NEC  operational  capability: 
levels  1  (less  mature)  to  level  5  (more  mature). 


-  Defines  5 
approaches  to 
C2  associated 
with  each  level. 


unconstrained  — - 


Si 


ss*- 


broad 


none 


fo 


VCy, 


& 


broad 


C2  Approaches 

i - Edge  C2  □ 

-Collaborative  C2  □ 

-Coordinated  C2  G 
De-Conflicted  C2  Q 
-Conflicted  C2  H 
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Background:  N2C2M2 


■  NATO  NEC  C2  Maturity  Model 

-  Defines  5  levels  of  NATO  NEC  operational  capability: 
levels  1  (less  mature)  to  level  5  (more  mature). 


-  Defines  5 
approaches  to 
C2  associated 
with  each  level. 


Level  4  Example 


tightly  constrained 

none 


broad 


unconstrained  — - 


2X3- 


broad 


C 

o 

8 

1 


C2  Approaches 

h - Edge  C2  □ 

-Collaborative  C2  □ 

-Coordinated  C2  G 
De-Conflicted  C2  Q 
-Conflicted  C2  H 
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Background:  N2C2M2 


■  NATO  NEC  C2  Maturity  Model  (2): 

-  More  maturity  delivers: 

■  More  effectiveness 

■  More  efficiency 

■  More  agility 

■  Positive  impact  in  intermediate  NCW  value-chain 

Variables,  such  as,  Quality  of  Individual  and  Shared  Information, 
Quality  of  Individual  and  Shared  Awareness  and  Understanding  and 
Self-Synchronization  * 


*  For  detailed  mapping  between  C2  CRM  variables  and  ELICIT  refer  to: 

MANSO,  Marco,  and  Paulo  NUNES.  ELICIT  and  the  Future  C2:  Theoretical  Foundations  for  the 
Analysis  of  ELICIT  Experiments.  Paper  presented  at  the  13th  ICCRTS,  Seattle,  USA,  2008 
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Background:  ELICIT 


■  ELICIT :  Experimental  Laboratory  for  the  Investigation 
of  Collaboration,  Information-sharing,  and  Trust. 

-  An  experimentation  environment  supported  by 
software  tools  and  instructions/procedures 

-  Provides  a  simple  (albeit  rich)  and  collaborative 
network-centric  environment  for  participating 
individuals 

■  Sponsor 

-  U.S.  DoD  Command  and  Control  Research  Program 
(www.dodccrp.org) 


Source:  Alberts  et.al.  “Assessing  Network  Centric  Operations  The  Challenge  of  NEC  C2,  A  Tutorial”, 
presentation  at  the  NCW-2009,  Washington,  DC,  January  27,  2009. 
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Background:  ELICIT 


Based  on:  Alberts  et.al.  “Assessing  Network  Centric  Opei 
Challenge  of  NEC  C2,  A  Tutorial”,  presentation  at  the  NC 
Washington,  DC,  January  27,  2009. 
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Experimentation:  Early  Expectations 


Main  hypotheses  for  validation: 


■  [1]  For  a  complex  endeavor ,  higher  collective  C2 
maturity  approaches  are  more  effective. 

■  [2]  For  a  given  level  of  effectiveness,  higher  collective 
C2  maturity  approaches  are  more  efficient. 

■  [3]  Higher  collective  C2  maturity  approaches  are  more 
agile. 
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Experimentation:  Early  Expectations 


Additional  hypotheses: 

■  Higher  collective  C2  maturity  approaches  exhibit  increased/better 
levels  of: 

[4]  Quality  of  Individual  and  Shared  Information; 

[5]  Quality  of  Individual  and  Shared  Awareness  and 
Understanding; 

[6]  Self-Synchronization  (at  cognitive  level); 

Than:  lower  collective  C2  maturity  approaches. 

■  [7]  Organizations  require  a  minimum  level  of  maturity  to  be 
effective  in  ELICIT. 

■  [8]  Increasing  the  degree  of  difficulty  in  ELICIT  require 

organizations  to  increase  their  level  of  maturity  to 
maintain  effectiveness  in  ELICIT. 
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Experimentation:  Model  and  Key-vars 
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d  Key-vars 
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Name 

Description 

Network 

Characteristics  and 
Performance 

Allow  or  restrict  interactions  between: 

-  subjects  and  teams. 

-  subjects  and  websites. 

This  variable  affects  Pl-C  and  Dl-C. 

Information 

Sharing  and 
(incentives  for) 
Collaboration 

Control:  predefined  server  distributions  of  all  factoids  to 
subjects  (in  three  waves). 

Influence:  distribution  of  information  as  a  result  of  human 
sharing  and  posting  (human  will) 

We  will  attempt  to  induce  /  influence  collaborative  behavior 
by: 

-  defining  collective  or  isolated  goals 

-  set  individual  and  collective  decision  rights  (see  ADR) 

See  paper  for  further  notes  on  Individual  and  Team 
Characteristics. 

This  variable  affects  Pl-C  and  Dl-C. 

Allocation  of 
Decision  Rights 

Decision  rights  will  be  allocated  according  to  the  C2 
Approach  to  implement: 

-  Distributed  for  higher  maturity  approaches; 

-  None  /  (de)centralized  for  lower  maturity  approaches. 

This  variable  is  a  C2  dimension. 
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Experimentation:  Design 


Common  aspects  for  all  approaches: 


■  Entities:  4  TEAMS  and  1  SINGLE  ENTITY,  except  EDGE  with  17 
ENTITIES 

■  Context:  complex  endeavor  with  two  or  more  force  elements  (entities) 
present  with  overlapping  intents;  operating  in  the  same  ‘space’  and  time; 
and,  an  entity  actions  may  conflict  with  those  taken  by  another  entities.” 
(Alberts  and  Hayes  2007). 

■  Scenario:  future  terrorist  attack 

■  Task:  identify  the  “who”,  “what”,  “where”  and  “when”  of  the  attack  within  a 
specific  timeframe. 

■  Information  Sharing  Capabilities:  share,  post  and  pull  actions.  High 
maturity  approaches  will  be  enriched  with  more  options  (see  next). 

■  Collaborative  Capabilities:  ability  to  provide  “assessment”  of  importance 
(relevance)  and/or  trustworthiness  of  a  factoid. 

■  Resource  Contention:  subject  hoarding  of  relevant  information  is 
considered  as  a  conflict.  Cognitive  efforts  are  required. 
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Experimentation:  CONFLICTED  model 


SUCESS  CRITERION 

Each  Team  pursues  independent  goals. 

Success  occurs  if  each  Team  leader  finds  the  correct  solution  to  his 
problem  space. 
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vs 


Experimentation:  DE-CONFLICTED  model 


ELICIT  Configuration  for  De-conflicted  C2  Approach 

-  Instructions  as  per  ELICIT  Hierarchy  Baseline 

Deconflictor 


Who  Team 


Where  Team 


Legend: 

O  Deconflictor 

o  Team  leader 
o  Team  member 


Who 

What 

When 

Where 

Web  Site 

Web  Site 

Web  Site 

Web  Site 

SUCESS  CRITERION 


Each  Team  pursues  independent  goals. 

Success  occurs  if  each  Team  leader  finds  the  correct  solution  to  his 
problem  space. 
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Experimentation:  COORDINATED  model 


ELICIT  Configuration  for  Coordinated  C2  Approach 

Configuration  similar  to  Hierarchy 
Instructions  define  role  of  coordinator 

Coordinator 


Who  Team 


Legend: 

O  Coordinator 
o  Team  leader 
o  Team  member 


Who 

What  Web 

Where 

When 

Web  Site 

Site 

Web  Site 

Web  Site 

SUCESS  CRITERION 

Organization  success  depends  on  the  Coordinator  finding  the  correct 
solution. 
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Experimentation:  COLLABORATIVE  model 


ELICIT  Configuration  for  Collaborative  C2  Approach 

Players  have  access  to  all  websites 

Feature  added  that  allows  players  to  evaluate 

factoids  and  share  /post  evaluations  Coordinator/ 

Instructions  define  role  of  coordinator  Facilitator 


Legend: 

o  Coordinator 
o  Team  leader 
o  Team  member 


Who  Team 


Where  Team 


SUCESS  CRITERION 

Coordinator  finding  the  correct  solution  to  all  problem  spaces  OR  Team 
leaders  finding  the  correct  solution  to  their  problem  space. 
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Experimentation:  EDGE  model 


/ 


ELICIT  Configuration  for  Edge  C2  Approach 


SUCESS  CRITERION 

Organization  success  depends  on  the  individuals’  IDs  plurality  being 
correct  in  each  problem  space. 
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Analysis:  Experiments  Baseline 


18  valid  runs  performed  with  human  subjects: 

■  3  runs  for  CONFLICTED 

■  4  runs  for  DE-CONFLICTED 

■  4  runs  for  COORDINATED 

■  4  runs  for  COLLABORATIVE 

■  3  runs  for  EDGE 

Usually,  a  group  of  17  subjects  was  used  to  perform  two  runs.  A  test  run  (15  to  30  min) 
was  always  conducted  prior  to  first  real  run. 

This  means: 

■  About  150  military  cadets  participated  in  ELICIT  runs. 

■  About  1 0  hours  of  data  to  analyze 

■  9  979  actions,  comprising: 

2  290  shares,  1  880  posts,  4  979  pulls  and  712  IDs 

■  Software  Analysis  Tool:  more  than  50K  SLOC 
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Analysis:  Overview 


■  Information  Domain 

■  Interactions  and  Social  Domain 

■  Cognitive  Domain 

■  Measures  of  Merit 
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Analysis:  Information  Domain 
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Analysis:  Information  Domain 
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Analysis:  Interactions  and  Social  Domain 


Nature  and  quantity  of  Interactions: 
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IN  and  OUT  Flows  (deviation  from  average):  per  role  and  per  approach 


O  CONFLICTED  (TLs) 

•  CONFLICTED  (TMs) 

O  DECONFLICTED  (IB/TLs) 

•  DECONFLICTED  (TMs) 

O  COORDINATED  (CTC/TLs) 

•  COORDINATED  (TMs) 

O  COLLABORATIVE  (CTC/TLs) 
O  COLLABORATIVE  (TMs) 


IN  DEV s  =  (nbr  _shares _receiveds  +  nbr  _ pulls s  )  -  (Average  nbr  shares _received  +  Average nbr  _  pulls) 
OUT  _  DEVS  =  (nbr  _  shares  _  sents  +  nbr  _  posts s  )  -  (Average _  nbr  _  shares  _  sent  +  Average _  nbr  _  posts ) 
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Analysis:  Interactions  and  Social  Domain 


Quality  of  Interactions 
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Analysis:  Cognitive  Domain 


Correct  Understanding 
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Analysis:  Cognitive  Domain 
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Mu  Analysis:  MoM 


Organization  Effectiveness 


Effectiveness  =Z°  .25  *  Correct _ansi 

i 

Correct_ansi  is  1 .0  if  correct  answer  is  provided  and  0.0  otherwise 
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Analysis:  MoM 
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Analysis:  MoM 


Time  Efficiency  (normalized) 
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Analysis:  MoM 


Effort  Efficiency  (normalized) 


Efficiency effort  =  Effectiveness 
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Conclusions 


Domain/ Variable 
Assessed 
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Conclusions 


kX  f 

Results  are  consistent  with  model  expectations  (in  overall): 

[4]  Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of  Quality 

of  Individual  and  Shared  Information  than  lower  collective  C2  maturity 
approaches. 

■  OK 

[5]  Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of  Quality 

of  Individual  and  Shared  Awareness  and  Understanding  than  lower  collective  C2 
maturity  approaches. 

■  OK  -  except  for  Coordinated 

[6]  Higher  collective  C2  maturity  approaches  exhibit  increased/better  levels  of  Self- 
Synchronization  (at  cognitive  level)  than  lower  collective  C2  maturity  approaches. 

■  OK 
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Conclusions 


Results  are  consistent  with  model  expectations  (in  overall): 

[1]  For  a  complex  endeavor,  higher  collective  C2  maturity  approaches 
are  more  effective. 

■  OK  -  except  for  EDGE 

[2]  For  a  given  level  of  effectiveness,  higher  collective  C2  maturity 
approaches  are  more  efficient. 

■  OK  -  except  for  EDGE  (with  a  high-deviation) 

[3]  Fligher  collective  C2  maturity  approaches  are  more  agile . 

NOT  Covered 

Agile  C2  is  a  novel  concept  under  the  analysis  of  SAS-085.  This 
hypothesis  will  be  considered  in  future  research  work. 
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Conclusions 


Results  are  consistent  with  model  expectations  (in  overall): 

[7]  Organizations  require  a  rninimum  level  of  maturity  to  be  effective 
ELICIT. 

Considering  current  dataset,  requisite  maturity  in  ELICIT  is 

COLLABORATIVE  or  COORDINATED  (having  a  proper  CTC). 

[8]  Increasing  the  degree  of  difficulty  in  ELICIT  require 
organizations  to  increase  their  level  ofjjiaturity  to  maintain 
effectiveness  in  ELICIT. 

NOT  Covered 

There  is  no  sufficient  data  (factoid  set  2  trials)  to  test  this  hypothesis. 
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Thank  You  for  Your  Attention 

Questions? 
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